Rapid identification of inflammatory arthritis and associated adverse events following immune checkpoint therapy: a machine learning approach

Introduction: Immune checkpoint inhibitor-induced inflammatory arthritis (ICI-IA) poses a major clinical challenge to ICI therapy for cancer, with 13% of cases halting ICI therapy and ICI-IA being difficult to identify for timely referral to a rheumatologist. The objective of this study was to rapidly identify ICI-IA patients in clinical data and assess associated immune-related adverse events (irAEs) and risk factors.

Methods: We conducted a retrospective study of the electronic health records (EHRs) of 89 patients who developed ICI-IA out of 2451 cancer patients who received ICI therapy at Northwestern University between March 2011 to January 2021. Logistic regression and random forest machine learning models were trained on all EHR diagnoses, labs, medications, and procedures to identify ICI-IA patients and EHR codes indicating ICI-IA. Multivariate logistic regression was then used to test associations between ICI-IA and cancer type, ICI regimen, and comorbid irAEs.

Results: Logistic regression and random forest models identified ICI-IA patients with accuracies of 0.79 and 0.80, respectively. Key EHR features from the random forest model included ICI-IA relevant features (joint pain, steroid prescription, rheumatoid factor tests) and features suggesting comorbid irAEs (thyroid function tests, pruritus, triamcinolone prescription). Compared to 871 adjudicated ICI patients who did not develop arthritis, ICI-IA patients had higher odds of developing cutaneous (odds ratio [OR]=2.66; 95% Confidence Interval [CI] 1.63-4.35), endocrine (OR=2.09; 95% CI 1.15-3.80), or gastrointestinal (OR=2.88; 95% CI 1.76-4.72) irAEs adjusting for demographics, cancer type, and ICI regimen. Melanoma (OR=1.99; 95% CI 1.08-3.65) and renal cell carcinoma (OR=2.03; 95% CI 1.06-3.84) patients were more likely to develop ICI-IA compared to lung cancer patients. Patients on nivolumab+ipilimumab were more likely to develop ICI-IA compared to patients on pembrolizumab (OR=1.86; 95% CI 1.01-3.43).

Discussion: Our machine learning models rapidly identified patients with ICI-IA in EHR data and elucidated clinical features indicative of comorbid irAEs. Patients with ICI-IA were significantly more likely to also develop cutaneous, endocrine, and gastrointestinal irAEs during their clinical course compared to ICI therapy patients without ICI-IA.

1 Introduction

Immune checkpoint inhibitors (ICIs) have become a pillar of cancer therapy, with demonstrated efficacy in many malignancies (1–9). ICIs are antibodies that antagonize checkpoints of T-cell development, enabling tumor-reactive T-cells to attack cancer cells (1, 2, 8, 10, 11). There are currently two major classes of ICIs. The first, approved in 2011, targets cytotoxic T-lymphocyte antigen 4 (CTLA-4). The second, first approved in 2014, targets programmed cell death protein 1 (PD-1) or its counterpart, programmed cell death ligand 1 (PD-L1). These ICIs may be used alone or combined with each other and other cancer therapies and are approved for a wide panel of cancers. An estimated 44% of US cancer patients are eligible for ICI therapy (4–7, 9, 12, 13).

However, while ICIs are effective anti-cancer agents, checkpoint blockade is associated with development of immune-related adverse events (irAEs) that affect a wide spectrum of organ systems (14–17). IrAEs can pose a significant barrier to ICI usage. They can prevent patients from continuing ICIs, diminish patient quality of life, and in severe cases, lead to death (3, 12, 13, 18). There are two major deficits in our current understanding of irAEs and our ability to care for patients with irAEs. First, irAEs are difficult to identify both in the patient care setting and for research studies. Outside of large clinical trials, many published studies are single-site with small cohorts identified and characterized by labor-intensive chart review (19–24). Second, we currently have a very limited ability to predict which patients will develop irAEs. Rheumatologic irAEs, including ICI-induced inflammatory arthritis (ICI-IA), epitomizes both of these limitations (24–35). ICI-IA has been recognized as an irAE for less than a decade with case studies and small cohort descriptions first appearing in the literature in 2017 (24–26). With a reported prevalence of 1-7%, it is relatively rare (36). However, it can have significant impact for patients with 12-13% of cases resulting in termination of ICI therapy (34, 35). Previous studies have described ICI-IA affecting the knees or small joints, tending to be seronegative for anti-cyclic citrullinated peptide (CCP) antibodies and rheumatoid factor, and most commonly treated with corticosteroids with elevation to disease-modifying antirheumatic drugs (DMARDs), tumor necrosis factor (TNF) inhibitors, and interleukin (IL)-6 receptor inhibitors (24–27, 29, 32, 37). Though development of ICI-IA cannot be well predicted, in a previous study, melanoma and genitourinary cancer as well as receiving combination ICI therapy were found to be associated with ICI-IA development compared to lung cancer and PD-1 monotherapy, respectively (38).

Prompt identification of patients with ICI-IA is essential for providing rapid referrals for effective clinical care and for performing research into the etiology of ICI-IA. However, ICI-IA is difficult to identify in clinical data, owing to its rarity, lack of dedicated diagnosis code, and heterogeneous presentation (24–32, 35). Furthermore, ICI-IA’s typically lower severity compared to other irAE such as myocarditis or pneumonitis means that patients may be less likely to be seen by a rheumatologist for their symptoms. While there have been numerous studies published on ICI-IA, ICI-IA cohort definition has primarily relied on manual identification of these patients by rheumatologists and/or filtering for patients who have been seen by a rheumatologist or been prescribed specific immunomodulatory drugs (26, 27, 29, 32, 34, 38). As a result, cohorts have remained relatively small and single site and run the risk of missing many ICI-IA patients who may have had a less severe presentation or other barriers to seeing a rheumatologist. Additionally, while many studies on ICI-IA include description of comorbid irAEs (26–28, 32), no previous study has tested the association of ICI-IA with these comorbid irAEs compared to a control cohort of ICI patients who did not develop arthritis, making it difficult to understand if the prevalence of comorbid irAEs experienced by ICI-IA patients is statistically different from the rest of the ICI population.

To address these fundamental gaps in knowledge and facilitate identification of ICI-IA and associations with the development of ICI-IA, we developed an electronic health record (EHR)-based machine learning strategy to rapidly identify patients who have possible ICI-IA and discover hidden features associated with ICI-IA. With the wide adoption of EHRs in inpatient and ambulatory settings (39), EHR data and data modeling strategies present an opportunity to develop tools to identify ICI-IA, as well as uncover clinical features of ICI-IA or associated conditions that may not be immediately obvious. Many previous studies have demonstrated success in identifying conditions including hypertension, stroke, systemic lupus erythematosus, asthma, and leukemia (40–44). To investigate the relationship between ICI-IA and other irAEs elucidated by our machine learning model, we also fully adjudicated a control cohort of 871 patients receiving ICI therapy for all irAEs, and used this control cohort to examine the association between the development of ICI-IA and other comorbid irAEs as well as cancer type and ICI regimen.

2 Materials and methods

2.1 Patient population and data source

The study population included patients seen in the Robert H. Lurie Comprehensive Cancer Center at Northwestern Medicine (NM), a large healthcare system providing inpatient, outpatient, and specialty care throughout Chicago and Northern Illinois. Data was acquired from the Northwestern Medicine Enterprise Data Warehouse (NMEDW), NM’s clinical research database containing data on over 10 million patients as of October 2023. This study was governed by the Northwestern University institutional review board, protocol #STU00210502 and STU00206779.

We retrospectively identified an ICI cohort of all patients aged 18 to 99 with a diagnosis of cancer (melanoma, renal cell carcinoma, non-small cell and small cell lung carcinoma, urothelial cancer, head and neck cancer, gastric cancer, colon cancer, liver cancer, cervical cancer, uterine cancer, breast cancer, Hodgkin’s lymphoma, Merkel cell carcinoma, rectal cancer, prostate cancer, esophageal cancer, leukemia, or lymphoma) who received at least one dose of ICI therapy (pembrolizumab, nivolumab, cemiplimab, atezolizumab, avelumab, durvalumab, ipilimumab, or tremilumumab) between March 1, 2011 and January 1, 2021 (Figure 1A). Cancer was identified in the NMEDW by International Classification of Disease-9^th revision-Clinical Modification (ICD-9-CM) and ICD-10-CM diagnosis codes. ICIs were identified in the NMEDW by a regular expression search for the generic and brand name in the medication data table (Supplementary Table 1). Sex, race, and ethnicity were gathered from patient demographic data present in the NMEDW. Prior autoimmune diseases were collected from NMEDW diagnosis codes (Supplementary Table 2).

Figure 1

Figure 1 Methods diagram. (A) Manual adjudication of ICI cohort (N=2451 patients) for immune checkpoint inhibitor-induced inflammatory arthritis (ICI-IA) cases (N=89 patients). From the ICI cohort, 871 random patients without ICI-IA (ICI-NoArthritis) were adjudicated for all irAE. (B) Electronic health record data extraction of all diagnosis (ICD-9-CM, ICD-10-CM), medication (RxNorm), laboratory test (LOINC), and procedure (CPT) codes. Individual code occurrences were modified to specify whether they occurred before or after ICI initiation, and dichotomized to presence/absence of the code. Data was extracted for the full ICI cohort of 2451 patients. (C) Logistic regression (LR) and random forest (RF) machine learning models were trained on the EHR codes to identify ICI-IA. (D) Feature importance was analyzed to characterize ICI-IA patients in the EHR. Multivariate logistic regression was used to calculate odds ratios for development of ICI-IA given cancer and ICI regimen, as well as development of non-arthritis irAEs given ICI-IA versus ICI-noIA.

2.2 Adjudication of ICI-induced inflammatory arthritis and statistical test control cohorts

Patient charts in NM’s Epic EHR were manually reviewed (SDT) with clinician guidance (CG, JL, AK, JS) to identify a case cohort of patients with ICI-IA (Figure 1A). We reviewed the charts of all patients who had the keywords: “arthritis”, “arthralgia”, or “joint” in the assessment and plan section of any clinical notes written after the patient received their first ICI dose. Cases were classified with ICI-IA if the patient had de novo joint pain/arthralgia/arthritis (no history of arthritis or presentation different from what the patient has experienced in the past), and an oncologist or rheumatologist noted suspicion of the presentation being secondary to ICI therapy. Case status, date of ICI-IA onset, cancer, ICI regimen, joint involvement, rheumatoid factor, anti-CCP, anti-nuclear antibodies (ANA), and treatment for ICI-IA were recorded in a REDCap database (45). Cases were additionally reviewed for other irAEs the patients experienced – cutaneous AEs, thyroid dysfunction, hypophysitis or adrenal insufficiency, diabetes, hepatic AEs, diarrhea, colitis, pneumonitis, cardiovascular AEs, and encephalitis. They were classified with an irAE if an oncologist or the relevant specialist noted suspicion of the presentation being secondary to ICI therapy without other likely etiologies. See Supplementary Methods for details.

The remaining patients without ICI-IA in the full ICI cohort were used as controls for our machine learning models. To compare irAE associations with ICI-IA, a random sample of 871 patients without ICI-IA (ICI-NoArthritis) from the overall ICI cohort of 2451 were chart reviewed for all irAEs (same irAEs and classification threshold as above) to serve as the control cohort for the statistical tests (SDT, GMP, KJR, CDM, JDJ, JT, KV, PD, SM, UR) (Figure 1A). Following initial adjudication, SDT reviewed a random sample of 10% of the charts and found greater than 90% agreement.

2.3 EHR code selection for machine learning

We collected all EHR clinical codes for every patient in our ICI cohort: ICD-9-CM and ICD-10-CM diagnosis codes, Logical Observation Identifiers Names and Codes (LOINC) laboratory codes, Unified Medical Language System (UMLS) RxNorm medication codes, and Current Procedural Terminology (CPT) procedure codes (Figure 1B). To determine if a machine learning model could select sensible ICI-IA-relevant codes and discover new predictors of ICI-IA, we input all EHR codes into our models. ICD-9-CM diagnosis codes were translated to ICD-10-CM to prevent duplication of diagnosis codes, using the concept relationships in the Observational Medical Outcomes Partnership (OMOP) common data model vocabulary tables (46). For each code, we added a temporal modifier specifying whether the code occurred before or after ICI initiation. We then dichotomized each to presence or absence of the code.

2.4 Machine learning to identify ICI-induced inflammatory arthritis in EHR data

The EHR codes were fed into Logistic Regression with Ridge (L2) penalty and Random Forest machine learning models to classify patients who experienced ICI-IA from the ICI cohort (Figure 1C). These models were selected for their capacity to provide clinically interpretable models. We bootstrapped model development 100 times to calculate model performance metrics with 95% confidence intervals and consensus code contribution. For each round of model development (each 1 of 100 bootstrap rounds), we used a random 50/50 training-testing split, stratified for consistent case/control proportions in the training and test sets. Cases were up-sampled during cross-validation and model training to balance the low ratio of ICI-IA cases to ICI controls (47). Five-fold cross-validation with 25 iterations was used to optimize model parameters in the training set. The test set was left untouched for performance evaluation. Models were evaluated using area under the receiver operating characteristic curve (AUROC). Accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were measured for all models, optimizing Youden’s J to balance sensitivity and specificity (48). EHR code contribution was calculated from the logistic regression beta coefficients and the random forest feature importance.

2.5 Key EHR codes in the ICI-induced inflammatory arthritis machine learning model

Feature importance from the random forest model was analyzed for the key EHR codes used to identify ICI-IA, averaging feature importance across the 100 bootstrapped models – codes with higher feature importance contribute more to the model for identifying ICI-IA (Figure 1D). While not a direct equivalence, these codes could indicate clinical features potentially describing ICI-IA and allow us to determine if the models were using ICI-IA-relevant information to capture patients with ICI-IA. The Fisher Exact test was used to determine the association between ICI-IA and the codes. Models were trained on decreasing percentages of the top codes (50% - 0.003%) and performance compared to the full models.

2.6 Cancer type and immune checkpoint inhibitor association with ICI-induced inflammatory arthritis

Multivariate logistic regression was used to calculate unadjusted and adjusted odds ratios (ORs) of developing ICI-IA given cancer type and first ICI regimen (specific ICI drug) (Figure 1D). Lung cancer and pembrolizumab were used as reference groups for cancer type and ICI regimen, respectively, as they represented our largest cancer type and treatment regimen. Covariates included in the calculation of adjusted ORs were sex, age, race, and ethnicity, cancer type for ICI ORs and ICI regimen for cancer ORs. Cancer type was determined by ICD-9/10 codes and ICI was determined by regex search as described in the patient population section (Supplementary Table 1). Cancer determination by ICD code compared to chart review showed <10% discrepancy. An ICI regimen was determined combination ICI therapy if two different ICIs were infused on the same date.

2.7 Determining irAE associations with ICI-induced inflammatory arthritis

Multivariate logistic regression was used to calculate unadjusted and adjusted ORs of developing non-arthritis irAEs given ICI-IA, compared to ICI-NoArthritis control patients (Figure 1D). IrAE associations were tested at three timeframes relative to ICI-IA development: irAE development any time after ICI initiation, only irAEs occurring prior to ICI-IA development, and only irAEs occurring post ICI-IA development. In the ICI-NoArthritis controls, irAE were included at any time after ICI initiation. Covariates included in the calculation of adjusted ORs were sex, age, race, ethnicity, cancer type, first ICI regimen, and presence of autoimmune disease prior to ICI initiation.

2.8 Analytical software

Statistical analysis and machine learning was performed using R 4.2.2 and Python 3.9.7 with scikit-learn, imbalanced-learn, scipy, and statsmodels packages. Statistical test results were considered statistically significant where p < 0.05. Models are available on GitHub (https://github.com/stevetran99/ICI-IA_ML_Classification).

3 Results

3.1 ICI-induced inflammatory arthritis cohort clinical characteristics

From the NMEDW, 2451 patients with a diagnosis of cancer who received at least one ICI dose between March 1, 2011 and January 1, 2021 were identified. Expert-guided chart review identified 89 cases of ICI-IA based on clinical suspicion without other likely etiologies in the oncology and rheumatology clinical notes. There were 2362 patients without ICI-IA remaining from the ICI cohort to serve as machine learning controls, with 871 of those patients adjudicated for other irAE serving as controls for our association tests.

Table 1 presents the demographic and cancer characteristics of the ICI-IA cohort and machine learning control cohort. We saw no significant differences across primary demographic characteristics (sex, age, race, ethnicity). Lung cancer (N=34, 38%), melanoma (N=29, 33%), and kidney cancer (N=20, 23%) were the most common cancers in our ICI-IA cohort. Pembrolizumab (N=28, 32%), nivolumab (N=23, 26%), and combination nivolumab-ipilimumab (N=27, 30%) were the predominant first ICI regimens and the majority of patients had no prior autoimmune disease (N=87, 98%). ICI-IA involved the knees (N=42, 47%), hand (N=31, 35%) and shoulder (N=28, 31%) (Table 2). The knees (N=30, 34%) and hands (N=22, 25%) were the most common first joints affected. Twenty-seven patients had diffuse joint pain. Of patients with specific joint involvement mentioned (N=71, 80%), the median number of distinct joint locations affected was 2, range 1 to 7. Of those tested for markers of rheumatologic disease, 5 of 20 patients were positive for rheumatoid factor, 0 of 16 patients were positive for anti-CCP antibodies, 9 of 15 patients were positive for ANA (Table 2). Eighteen (20%) patients were seen by a rheumatologist. ICI-IA onset was a median of 20 weeks (IQR 8-45 weeks) after initiating checkpoint therapy. Primary treatment modalities for the ICI-IA symptoms were steroids (N=43, 48%), and NSAIDs (N=33, 37%). In 18 (20%) patients, ICI therapy was held or stopped entirely due to arthritis.

Table 1

Table 1 Demographic and clinical variables for cancer patients receiving Immune Checkpoint Inhibitor Therapy between March 1, 2011 and January 1, 2021.

Table 2

Table 2 Immune checkpoint inhibitor-induced inflammatory arthritis (ICI-IA) laboratory tests, joint involvement, and comorbid irAEs for our cohort of patients with ICI-IA.

3.2 EHR-based machine learning reliably identified ICI-induced inflammatory arthritis

Machine learning models were trained to identify ICI-IA in EHR data on the 89 ICI-IA cases and 2362 machine learning controls. We identified 32,682 EHR codes representing all diagnoses, procedures, labs, and medications, with the temporal modifiers for codes occurring before and after ICI initiation as variables for the models. Trained on the full code set, logistic regression modeling achieved an AUROC of 0.77 (95% Confidence Interval [CI] 0.73-0.82) while random forest modeling achieved an AUROC of 0.81 (95% CI 0.76-0.86) (Table 3). Training on decreasing percentages of the top codes derived from the random forest model showed consistently high performance for logistic regression and random forest models before dropping in performance below 31 codes (Figure 2). Our highest performing model, random forest with the top 31 codes, achieved an AUROC of 0.81 (95% CI 0.75-86), accuracy of 0.79 (95% CI 0.63-0.94), sensitivity of 0.71 (95% CI 0.54-0.88), specificity of 0.79 (95% CI 0.62-0.95), PPV of 0.13 (95% CI 0.03-0.22), and NPV of 0.99 (95% CI 0.98-0.99) (Table 3).

Table 3

Table 3 Logistic Regression and Random Forest machine learning models performance metrics for identifying ICI-induced inflammatory arthritis (ICI-IA).

Figure 2

Figure 2 Model performance (area under the receiver operating characteristic curve, AUROC) versus percentage of the top features used to develop random forest and logistic regression models. Models maintain high performance with decreasing percentage of top features included before dropping performance with fewer than 0.1% of features or 31 features (vertical dotted line).

3.3 Key EHR codes in the ICI-induced inflammatory arthritis model were potentially related to other irAEs

From the random forest model, we evaluated the 55 most important codes to elucidate clinical features relevant to identifying ICI-IA. Codes were categorized as ICI-IA relevant (codes matching elements of ICI-IA presentation, diagnosis, or management in the literature), potentially relevant to other irAEs, and those that were related to other medical history elements. Twenty-two were relevant to ICI-IA, including diagnosis codes for unspecified joint pain, knee pain, and unspecified osteoarthritis (osteoarthritis was likely a placeholder early in the diagnostic workflow); laboratory test and procedure codes for erythrocyte sedimentation rate (ESR), C-reactive protein (CRP), rheumatoid factor, and anti-CCP antibody tests; medication codes for prednisone and methylprednisolone. (‘ICI-IA’ codes in Figure 3). Sixteen were potentially relevant to other irAEs, including codes for endocrine disorder screening, thyroid function tests, cortisol labs, medication codes for triamcinolone, and diagnosis codes for pruritus and myositis (‘irAE’ codes in Figure 3). Seventeen were related to other medical history elements such as COVID-19 or chemotherapy administration (‘Other’ codes in Figure 3).

Figure 3

Figure 3 Key EHR codes in the machine learning models and association with ICI-induced inflammatory arthritis (ICI-IA). Left: the random forest model’s feature importance for identifying ICI-IA patients. Right: odds of the patient having an EHR code if they developed ICI-IA versus if they did not, by Fisher Exact test. Error bars are 95% confidence intervals. ‘ICI-IA’ codes are those directly relevant to ICI-IA. ‘irAE’ codes are those potentially describing other irAEs. ‘Other’ codes are those describing other parts of the patient medical history. The top codes are predominantly ICI-IA relevant codes, with a high concentration of relevant codes occupying the topmost importance. The top irAE related codes are endocrine (cortisol, thyroid function tests, estradiol), myositis, and cutaneous (medication order for triamcinolone and pruritus). The majority of the top codes are positively associated with ICI-IA. Codes are labeled by name, code vocabulary, code, and temporal modifier (before or after ICI therapy initiation).

3.4 Melanoma, renal cell carcinoma, and combination nivolumab-ipilimumab therapy were associated with development of ICI-induced inflammatory arthritis

Melanoma, OR = 1.99 (95% CI 1.08-3.65) and renal cell carcinoma patients, OR = 2.03 (95% CI 1.06-3.84), had higher odds of developing ICI-IA compared to lung cancer patients, adjusting for sex, age, race, ethnicity, and first ICI regimen. Combination nivolumab-ipilimumab treatment was associated with higher odds of developing ICI-IA, OR = 1.86 (95% CI 1.01-3.43), compared to patients who received pembrolizumab, adjusting for sex, age, race, ethnicity, and cancer type (Table 4).

Table 4

Table 4 Odds of developing ICI-induced inflammatory arthritis (ICI-IA) given cancer and first ICI.

3.5 ICI-induced inflammatory arthritis was associated with development of cutaneous, endocrine, and gastrointestinal irAEs

We further explored the key irAE related EHR codes from our ICI-IA machine learning model. The irAEs most commonly documented as co-occurring with ICI-IA were cutaneous irAE: pruritus, rash, and vitiligo (N=38); gastrointestinal irAE: diarrhea and constipation (N=26) and colitis (N=9); endocrine irAE: thyroid dysfunction (N=16), hypophysitis and adrenal insufficiency (N=10), and diabetes (N=1); and hepatic irAE: transaminitis and hepatitis (N=8). Four ICI-IA patients experienced pneumonitis. Pericarditis and encephalitis were documented in one separate patient each. Twenty-one ICI-IA patients had no other documented irAE.

ICI-IA patients had higher odds of having any additional documented irAE, odds ratio (OR) = 2.53 (95% CI 1.49-4.31), compared to ICI-NoArthritis patients, adjusting for sex, age, race, ethnicity, cancer type, first ICI regimen, and presence of autoimmune disease prior to ICI therapy (Table 5). They specifically had higher odds of experiencing cutaneous irAE, OR = 2.66 (95% CI 1.63-4.35), endocrine irAE, OR = 2.09 (95% CI 1.15-3.80), and gastrointestinal irAE, OR = 2.88 (95% CI 1.76-4.72) (Table 5).

Table 5

Table 5 Odds ratio (OR) of developing irAEs given ICI-induced inflammatory arthritis (ICI-IA).

In unadjusted models, ICI-IA patients had higher odds of developing cutaneous irAE, OR = 1.65 (95% CI 1.00-2.72), and gastrointestinal irAE, OR = 1.76 (95% CI 1.07-2.90), prior to ICI-IA development compared to the ICI-NoArthritis control developing these irAEs at any time. ICI-IA patients had higher odds of developing thyroid dysfunction, OR = 2.01 (95% CI 1.01-4.00), post ICI-IA development compared to the ICI-NoArthritis control developing thyroid dysfunction at any time (Table 5). However, these temporal associations were no longer statistically significant when adjusting for sex, age, race, ethnicity, cancer type, first ICI regimen, and presence of prior autoimmune disease.

4 Discussion

Using an ICI cohort of 2451 patients, 89 of which had ICI-IA, we developed an EHR-based machine learning model that could rapidly identify patients who had possible ICI-IA with AUROCs of 0.80-0.81 and accuracy of 0.79-0.80. Our machine learning model captured key EHR codes relevant to ICI-IA as well as codes indicative of cutaneous and endocrine irAEs. On further investigation of these irAE related codes, we found that ICI-IA was associated with development of additional cutaneous, endocrine, and gastrointestinal irAEs independent of cancer type and ICI regimen.

To develop our EHR-based machine learning model of ICI-IA, we created a cohort of 2451 patients who had cancer and received an ICI, including 89 ICI-IA patients, and 871 patients without ICI-IA, who were fully adjudicated for irAEs by manual chart review. We expanded on an approach described by Thangaraj et al. and used all diagnosis, laboratory test, procedure, and medication codes in the EHR to develop an ICI-IA identification algorithm (43). Both logistic regression and random forest machine learning models performed well on the full code set (32,682 codes) and maintained high performance while reducing the number of EHR codes used to develop the models – at 31 codes, logistic regression had an AUROC of 0.80 (95% CI 0.75-0.86) while random forest had an AUROC of 0.81 (95% CI 0.75-0.86) However, both models had low PPVs of 0.13, despite good AUROC and accuracy. This is an inherent limitation of PPV in rare conditions such as ICI-IA, with a prevalence only 3.6% in our ICI cohort. This also speaks to the purpose of our ICI-IA model as a filter for potential ICI-IA cases for clinician verification rather than as a replacement of clinical expertise.

A major advantage of this approach compared to traditional manual selection of model features is that our EHR model provided information on codes useful for identifying ICI-IA patients. While these codes are not a direct equivalence, they were indicators of clinical features that could describe ICI-IA. Both models maintained high performance down to 0.1% of the total code set, suggesting the majority of information important for identifying ICI-IA patients was held by a small fraction of EHR codes. We evaluated the top codes from the random forest model to understand the key predictive elements. Twenty-two of the 55 top codes were determined to be relevant to ICI-IA including lab and procedure codes for ESR, CRP, and rheumatoid factor, diagnosis codes for joint pain, and medication codes for steroids. These factors are consistent with the ICI-IA literature regarding ICI-IA presentation, diagnostic tests, and therapy (27, 29, 37, 49). The density of ICI-IA relevant codes in the group of features with the highest feature importance – the top 8 codes were ICI-IA relevant codes – indicates that our strategy constructed a clinically sensible model to capture ICI-IA relevant information without manual guidance on which codes should be used, increasing the credibility of the other unexpected key EHR codes we found.

While many of the top codes identified by our models were similar to those identified by clinical experts in ICI-IA, 16 of the 55 top codes appeared related to other irAEs, suggesting a possible relationship between ICI-IA and other irAE. We found codes for thyroid stimulating hormone (TSH) and T3 thyroid function tests, cortisol tests, pruritus, and triamcinolone, all after ICI initiation. These codes indicated potential association of ICI-IA with endocrine and cutaneous irAE. Indeed the literature suggests comorbid cutaneous, endocrine, gastrointestinal, and other irAEs with development of ICI-IA (26, 27). However, to our knowledge we are the first to statistically associate that ICI-IA patients were significantly more likely than ICI patients without ICI-IA to also develop cutaneous, OR = 2.66 (95% CI 1.63-4.35), endocrine, OR = 2.09 (95% CI 1.15-3.80), and gastrointestinal irAEs, OR = 2.88 (95% CI 1.76-4.72), adjusting for sex, age, race, ethnicity, cancer type, first ICI regimen, and presence of autoimmune disease prior to ICI initiation. ICI-IA patients were also more likely to experience any other irAE, OR = 2.53 (95% CI 1.49-4.31). To our knowledge, this is the first study that has conducted tests of association of irAEs comparing ICI-IA to a control cohort of ICI patients without ICI-IA, with both cohorts fully adjudicated for irAEs. This confirms the results of previous studies (26, 30, 32, 38) and shows that patients with ICI-IA are actually experiencing higher rates of cutaneous, endocrine, and gastrointestinal irAE compared to the general ICI population. Additionally, our findings suggest that the association between ICI-IA and cutaneous, endocrine, and gastrointestinal irAEs is independent of cancer type, ICI regimen, and prior autoimmune disease.

Conducting temporal association tests for irAEs that occurred before and after ICI-IA, we found that ICI-IA patients were more likely to have developed cutaneous irAEs and gastrointestinal irAEs prior to ICI-IA, in an unadjusted model. Conversely, ICI-IA patients were more likely to develop thyroid dysfunction after ICI-IA. Our temporal association findings indicate a possible common phenotype of patients who receive ICI therapy, develop skin and/or gastrointestinal irAEs, followed by ICI-IA, and finally thyroid dysfunction. However, when adjusting for demographics, cancer type, ICI regimen, and prior autoimmune disease, these temporal associations were no longer statistically significant, suggesting that the temporal phenotypes may be dependent on cancer type and ICI regimen or specific to our cohort and will require larger cohort sizes and future study.

As part of our analysis, we recapitulated findings in the literature of associations between cancer and ICI regimen and development of ICI-IA. We found similar associations in our ICI-IA cohort to that observed previously by Cunningham-Bussel et al. (38) even with our differing inclusion protocol for ICI-IA, increasing our confidence in the representativeness of our ICI-IA and ICI cohorts. Melanoma patients, OR = 1.99 (95% CI 1.08-3.65), and renal cell carcinoma patients, OR = 2.03 (95% CI 1.06-3.84), had higher odds of developing ICI-IA compared to lung cancer patients, adjusting for sex, age, race, ethnicity, and first ICI regimen. We also found that patients whose first ICI regimen was combination nivolumab-ipilimumab had higher odds of developing ICI-IA, OR = 1.86 (95% CI 1.01-3.43), compared to patients who received pembrolizumab, adjusting for sex, age, race, ethnicity, and cancer type. We additionally found similar time elapsed from ICI initiation to ICI-IA development compared to the ICI-IA literature – median of 20 weeks (5 months) compared to 2.7 – 5 months (30, 34, 50).

Our study is not without limitations. Despite being larger than most published studies, our ICI-IA cohort is still relatively small (N=89) and constrained to a single site. This small cohort relative to the number of EHR codes could make training our machine learning models difficult. We mitigated this issue by performing feature reduction based on the feature importance provided by the initial random forest model and saw high-performance using only 0.1% of the total code set. Another limitation is that our ICI-IA adjudication was done by retrospective chart review, therefore relying on clinician documentation of ICI-IA, and included patients that were not seen by a rheumatologist. Thus, some of these patients may have arthralgia or polymyalgia rheumatica rather than arthritis. However, as we expand this study to further sites and patients, we will be able to better make this distinction. We chose to accept these limitations to capture a broader cohort of patients with potentially less severe ICI-IA or who had other barriers to seeing a rheumatologist. Our finding that only 18 of the 89 ICI-IA patients were seen by a rheumatologist further highlights the shortcomings of the current standard for defining ICI-IA cohorts and the importance of our model for identifying ICI-IA patients.

In summary, we developed a novel EHR-based machine learning algorithm that was able to identify ICI-IA patients in the EHR with high performance. This EHR algorithm could be adapted to other EHR systems to facilitate cohort definition for ICI-IA for multicenter research studies and to assist in clinical practice with recommending patients suspected to have ICI-IA to follow-up care with a rheumatologist. Our machine learning strategy revealed hidden relationships between ICI-IA and other irAE that were corroborated in association tests. It provided insight into the clinical features that were descriptive of ICI-IA, including features potentially pertaining to comorbid irAEs. By further examining these irAEs, we found that cutaneous, endocrine, and gastrointestinal irAEs were significantly more likely to be found in patients with ICI-IA compared to cancer patients receiving ICI therapy who did not experience ICI-IA, independent of cancer type and ICI regimen. We also found potential temporal relationships between ICI-IA and other irAEs. This indicates that other such temporal phenotypes may exist in patients who develop irAEs that could be captured in EHR data. Overall, further exploration of irAEs in EHR data and investigation of irAE temporal phenotypes may reveal leads for mechanistic studies of irAE development and improve care for these patients through more rapid identification of irAEs and timelier recommendation for follow-up care.

Data availability statement

The data analyzed in this study is subject to the following licenses/restrictions: The data used and analyzed during the current study are available from the corresponding author on reasonable request. Note that row level data is access controlled and cannot be provided publicly due to data use restrictions. Models are available on GitHub (https://github.com/stevetran99/ICI-IA_ML_Classification). Requests to access these datasets should be directed to TLW, t-walunas@northwestern.edu.

Ethics statement

The studies involving humans were approved by Northwestern University Institutional Review Board. The studies were conducted in accordance with the local legislation and institutional requirements. The ethics committee/institutional review board waived the requirement of written informed consent for participation from the participants or the participants’ legal guardians/next of kin because impracticality of consent and low risk of the study.

Author contributions

SDT: Data curation, Formal Analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Visualization, Writing – original draft, Writing – review & editing, Conceptualization. JL: Data curation, Writing – review & editing, Methodology. CG: Data curation, Methodology, Writing – review & editing. LVR: Data curation, Methodology, Writing – review & editing. JP: Data curation, Methodology, Writing – review & editing. GMP: Data curation, Writing – review & editing. KJR: Data curation, Writing – review & editing. CDM: Data curation, Writing – review & editing. JDJ: Data curation, Writing – review & editing. JT: Data curation, Writing – review & editing. KV: Data curation, Writing – review & editing. PVD: Data curation, Writing – review & editing. SM: Data curation, Writing – review & editing. UR: Data curation, Writing – review & editing. KT: Data curation, Writing – review & editing. NM: Investigation, Writing – review & editing. JLJ: Investigation, Writing – review & editing. YL: Investigation, Methodology, Writing – review & editing. AK: Investigation, Writing – review & editing. JS: Investigation, Writing – review & editing. TLW: Writing – review & editing, Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This project was supported by Grant Number R61-AR076824 from the National Institute of Arthritis and Musculoskeletal and Skin Diseases. SDT is supported by Grant Number 1F31LM014201 from the National Library of Medicine. JL is supported in part by Grant Number T32 AR007611-13 from the National Institute of Arthritis and Musculoskeletal and Skin Diseases. The Northwestern Medicine Enterprise Data Warehouse is supported by Grant Number UL1 TR001422-09.

Conflict of interest

TLW receives unrelated research funding from Gilead Sciences. AK is a strategic advisor for Datavant, Inc.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.​

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fimmu.2024.1331959/full#supplementary-material

Abbreviations

ANA, Anti-nuclear antibody; AUROC, Area under the receiver operating characteristic curve; CCP, Cyclic citrullinated peptide; CI, Confidence interval; CPT, Current Procedural Terminology; CRP, C-reactive protein; CTLA-4, Cytotoxic T-lymphocyte antigen 4; DMARD, Disease-modifying antirheumatic drug; HER, Electronic health record; ESR, Erythrocyte sedimentation rate; ICD, International Classification of Disease; ICI, Immune checkpoint inhibitor; ICI-IA, Immune checkpoint inhibitor-induced inflammatory arthritis; IL, Interleukin; irAE, Immune-related adverse event; LOINC, Logical Observation Identifiers Names and Codes; ML, Machine learning; NM, Northwestern Medicine; NMEDW, Northwestern Medicine Enterprise Data Warehouse; NPV, Negative predictive value; OMOP, Observational Medical Outcomes Partnership; OR, Odds ratio; PD-1, Programmed cell death protein 1; PD-L1, Programmed cell death ligand 1; PPV, Positive predictive value; TNF, Tumor necrosis factor; TSH, Thyroid stimulating hormone; UMLS, Unified Medical Language System.

References

1. Hodi FS, O'Day SJ, Mcdermott DF, Weber RW, Sosman JA, Haanen JB, et al. Improved survival with ipilimumab in patients with metastatic melanoma. New Engl J Med. (2010) 363:711–23. doi: 10.1056/NEJMoa1003466

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Poole RM. Pembrolizumab: first global approval. Drugs. (2014) 74:1973–81. doi: 10.1007/s40265-014-0314-5

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Garon EB, Rizvi NA, Hui R, Leighl N, Balmanoukian AS, Eder JP, et al. Pembrolizumab for the treatment of non–small-cell lung cancer. New Engl J Med. (2015) 372:2018–28. doi: 10.1056/NEJMoa1501824

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Larkin J, Chiarion-Sileni V, Gonzalez R, Grob JJ, Cowey CL, Lao CD, et al. Combined nivolumab and ipilimumab or monotherapy in untreated melanoma. New Engl J Med. (2015) 373:23–34. doi: 10.1056/NEJMoa1504030

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Postow MA, Chesney J, Pavlick AC, Robert C, Grossmann K, Mcdermott D, et al. Nivolumab and ipilimumab versus ipilimumab in untreated melanoma. New Engl J Med. (2015) 372:2006–17. doi: 10.1056/NEJMoa1414428

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Wolchok JD, Chiarion-Sileni V, Gonzalez R, Rutkowski P, Grob J-J, Cowey CL, et al. Overall survival with combined nivolumab and ipilimumab in advanced melanoma. New Engl J Med. (2017) 377:1345–56. doi: 10.1056/NEJMoa1709684

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Gandhi L, Rodríguez-Abreu D, Gadgeel S, Esteban E, Felip E, De Angelis F, et al. Pembrolizumab plus chemotherapy in metastatic non–small-cell lung cancer. New Engl J Med. (2018) 378:2078–92. doi: 10.1056/NEJMoa1801005

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Hargadon KM, Johnson CE, Williams CJ. Immune checkpoint blockade therapy for cancer: An overview of FDA-approved immune checkpoint inhibitors. Int Immunopharmacology. (2018) 62:29–39. doi: 10.1016/j.intimp.2018.06.001

CrossRef Full Text | Google Scholar

9. Haslam A, Prasad V. Estimation of the percentage of US patients with cancer who are eligible for and respond to checkpoint inhibitor immunotherapy drugs. JAMA Network Open. (2019) 2:e192535. doi: 10.1001/jamanetworkopen.2019.2535

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Walunas TL, Lenschow DJ, Bakker CY, Linsley PS, Freeman GJ, Green JM, et al. CTLA-4 can function as a negative regulator of T cell activation. Immunity. (1994) 1:405–13. doi: 10.1016/1074-7613(94)90071-X

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Agata Y, Kawasaki A, Nishimura H, Ishida Y, Tsubat T, Yagita H, et al. Expression of the PD-1 antigen on the surface of stimulated mouse T and B lymphocytes. Int Immunol. (1996) 8:765–72. doi: 10.1093/intimm/8.5.765

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Hellmann MD, Paz-Ares L, Bernabe Caro R, Zurawski B, Kim S-W, Carcereny Costa E, et al. Nivolumab plus ipilimumab in advanced non–small-cell lung cancer. New Engl J Med. (2019) 381:2020–31. doi: 10.1056/NEJMoa1910231

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Yau T, Kang Y-K, Kim T-Y, El-Khoueiry AB, Santoro A, Sangro B, et al. Efficacy and safety of nivolumab plus ipilimumab in patients with advanced hepatocellular carcinoma previously treated with sorafenib. JAMA Oncol. (2020) 6:e204564. doi: 10.1001/jamaoncol.2020.4564

PubMed Abstract | CrossRef Full Text | Google Scholar

14. González-Rodríguez E, Rodríguez-Abreu D. Immune checkpoint inhibitors: review and management of endocrine adverse events. Oncologist. (2016) 21:804–16. doi: 10.1634/theoncologist.2015-0509

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Wang DY, Salem JE, Cohen JV, Chandra S, Menzer C, Ye F, et al. Fatal toxic effects associated with immune checkpoint inhibitors: A systematic review and meta-analysis. JAMA Oncol. (2018). 4:1721–8. doi: 10.1001/jamaoncol.2018.3923

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Martins F, Sofiya L, Sykiotis GP, Lamine F, Maillard M, Fraga M, et al. Adverse effects of immune-checkpoint inhibitors: epidemiology, management and surveillance. Nat Rev Clin Oncol. (2019) 16:563–80. doi: 10.1038/s41571-019-0218-0

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Johnson DB, Chandra S, Sosman JA. Immune checkpoint inhibitor toxicity in 2018. JAMA. (2018) 320:1702. doi: 10.1001/jama.2018.13995

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Fuchs CS, Doi T, Jang RW, Muro K, Satoh T, MaChado M, et al. Safety and efficacy of pembrolizumab monotherapy in patients with previously treated advanced gastric and gastroesophageal junction cancer. JAMA Oncol. (2018) 4:e180013. doi: 10.1001/jamaoncol.2018.0013

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Gallan AJ, Alexander E, Reid P, Kutuby F, Chang A, Henriksen KJ. Renal vasculitis and pauci-immune glomerulonephritis associated with immune checkpoint inhibitors. Am J Kidney Dis. (2019), 74:853–6. doi: 10.1053/j.ajkd.2019.04.016

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Zen Y, Yeh MM. Hepatotoxicity of immune checkpoint inhibitors: A histology study of seven cases in comparison with autoimmune hepatitis and idiosyncratic drug-induced liver injury. Modern Pathology: Nat Publishing Group;. (2018), 31:965–73. doi: 10.1038/s41379-018-0013-y

CrossRef Full Text | Google Scholar

21. Wang Y, Abu-Sbeih H, Mao E, Ali N, Qiao W, Trinh VA, et al. Endoscopic and histologic features of immune checkpoint inhibitor-related colitis. Inflammatory Bowel Dis. (2018), 24:1695–705. doi: 10.1093/ibd/izy104

CrossRef Full Text | Google Scholar

22. Belliere J, Meyer N, Mazieres J, Ollier S, Boulinguez S, Delas A, et al. Acute interstitial nephritis related to immune checkpoint inhibitors. Br J Cancer. (2016), 115:1457–61. doi: 10.1038/bjc.2016.358

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Sanlorenzo M, Vujic I, Daud A, Algazi A, Gubens M, Luna SA, et al. Pembrolizumab cutaneous adverse events and their association with disease progression. JAMA Dermatol. (2015), 151:1206–12. doi: 10.1001/jamadermatol.2015.1916

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Calabrese C, Kirchner E, Kontzias K, Velcheti V, Calabrese LH. Rheumatic immune-related adverse events of checkpoint therapy for cancer: case series of a new nosological entity. RMD Open. (2017) 3:e000412. doi: 10.1136/rmdopen-2016-000412

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Belkhir r, Le Burel s, Dunogeant L, Marabelle A, Hollebecque A, Besse B, et al. Rheumatoid arthritis and polymyalgia rheumatica occurring after immune checkpoint inhibitor treatment. Ann Rheum Dis. (2017), 76:1747–50. doi: 10.1136/annrheumdis-2017-211216

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Cappelli LC, Gutierrez AK, Baer AN, Albayda J, Manno RL, Haque U, et al. Inflammatory arthritis and sicca syndrome induced by nivolumab and ipilimumab. Ann Rheumatic Diseases. (2017) 76:43–50. doi: 10.1136/annrheumdis-2016-209595

CrossRef Full Text | Google Scholar

27. Cappelli LC, Brahmer JR, Forde PM, Le DT, Lipson EJ, Naidoo J, et al. Clinical presentation of immune checkpoint inhibitor-induced inflammatory arthritis differs by immunotherapy regimen. Semin Arthritis Rheumatism. (2018), 48:553–7. doi: 10.1016/j.semarthrit.2018.02.011

CrossRef Full Text | Google Scholar

28. Inamo J, Kaneko Y, Takeuchi T. Inflammatory tenosynovitis and enthesitis induced by immune checkpoint inhibitor treatment. Clin Rheumatol. (2018) 37:1107–10. doi: 10.1007/s10067-018-4035-0

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Leipe J, Christ LA, Arnoldi AP, Mille E, Berger F, Heppt M, et al. Characteristics and treatment of new-onset arthritis after checkpoint inhibitor therapy. Open. (2018), 4:714. doi: 10.1136/rmdopen-2018-000714

CrossRef Full Text | Google Scholar

30. Arnaud L, Lebrun-Vignes B, Salem J-E. Checkpoint inhibitor-associated immune arthritis. Ann Rheumatic Diseases. (2019) 78:e68–e. doi: 10.1136/annrheumdis-2018-213470

CrossRef Full Text | Google Scholar

31. Roberts J, Smylie M, Walker J, Basappa NS, Chu Q, Kolinsky M, et al. Hydroxychloroquine is a safe and effective steroid-sparing agent for immune checkpoint inhibitor–induced inflammatory arthritis. Clin Rheumatol. (2019), 38:1513–9. doi: 10.1007/s10067-019-04451-2

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Braaten TJ, Brahmer JR, Forde PM, Le D, Lipson EJ, Naidoo J, et al. Immune checkpoint inhibitor-induced inflammatory arthritis persists after immunotherapy cessation. Ann Rheumatic Dis. (2019), 79:332–8. doi: 10.1136/annrheumdis-2019-216109

CrossRef Full Text | Google Scholar

33. Calabrese L, Mariette X. Chronic inflammatory arthritis following checkpoint inhibitor therapy for cancer: game changing implications. Ann Rheumatic Diseases. (2020) 79:309–11. doi: 10.1136/annrheumdis-2019-216510

CrossRef Full Text | Google Scholar

34. Chan KK, Tirpack A, Vitone G, Benson C, Nguyen J, Ghosh N, et al. Higher Checkpoint Inhibitor Arthritis Disease Activity may be Associated With Cancer Progression: Results From an Observational Registry. ACR Open Rheumatol. (2020) 2:595–604. doi: 10.1002/acr2.11181

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Ghosh N, Tiongson MD, Stewart C, Chan KK, Jivanelli B, Cappelli L, et al. Checkpoint inhibitor–associated arthritis: A systematic review of case reports and case series. JCR: J Clin Rheumatol. (2020). 00:1–6. doi: 10.1097/RHU.0000000000001370

CrossRef Full Text | Google Scholar

36. Cappelli LC, Gutierrez AK, Bingham CO, Shah AA. Rheumatic and musculoskeletal immune-related adverse events due to immune checkpoint inhibitors: A systematic review of the literature. Arthritis Care Res. (2017) 69:1751–63. doi: 10.1002/acr.23177

CrossRef Full Text | Google Scholar

37. Jeurling S, Cappelli LC. Treatment of immune checkpoint inhibitor-induced inflammatory arthritis. Curr Opin Rheumatol. (2020) 32:315–20. doi: 10.1097/BOR.0000000000000701

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Cunningham-Bussel A, Wang J, Prisco LC, Martin LW, Vanni KMM, Zaccardelli A, et al. Predictors of rheumatic immune-related adverse events and de novo inflammatory arthritis after immune checkpoint inhibitor treatment for cancer. Arthritis Rheumatol. (2022) 74:527–40. doi: 10.1002/art.41949

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Henry J, Pylypchuk Y, Searcy T, Patel V. Adoption of Electronic Health Record Systems among U.S. Non-Federal Acute Care Hospitals: 2008-2015. Washington DC: ONC Data Brief: Office of the National Coordinator for Health Information Technology (2016).

Google Scholar

40. Pacheco JA, Avila PC, Thompson JA, Law M, Quraishi JA, Greiman AK, et al. A highly specific algorithm for identifying asthma cases and controls for genome-wide association studies. AMIA Annu Symposium Proc / AMIA Symposium AMIA Symposium. (2009) 497–501.

Google Scholar

41. Teixeira PL, Wei W-Q, Cronin RM, Mo H, Vanhouten JP, Carroll RJ, et al. Evaluating electronic health record data sources and algorithmic approaches to identify hypertensive individuals. J Am Med Inf Assoc. (2017) 24:162–71. doi: 10.1093/jamia/ocw071

CrossRef Full Text | Google Scholar

42. Phillips CA, Razzaghi H, Aglio T, Mcneil MJ, Salvesen-Quinn M, Sopfe J, et al. Development and evaluation of a computable phenotype to identify pediatric patients with leukemia and lymphoma treated with chemotherapy using electronic health record data. Pediatr Blood Cancer. (2019) 66:e27876. doi: 10.1002/pbc.27876

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Thangaraj PM, Kummer BR, Lorberbaum T, Elkind MSV, Tatonetti NP. Comparative analysis, applications, and interpretation of electronic health record-based stroke phenotyping methods. BioData Min. (2020), 13:21. doi: 10.1186/s13040-020-00230-x

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Walunas TL, Ghosh AS, Pacheco JA, Mitrovic V, Wu A, Jackson KL, et al. Evaluation of structured data from electronic health records to identify clinical classification criteria attributes for systemic lupus erythematosus. Lupus Sci Med. (2021) 8:e000488. doi: 10.1136/lupus-2021-000488

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Informatics. (2009) 42:377–81. doi: 10.1016/j.jbi.2008.08.010

CrossRef Full Text | Google Scholar

46. Resnic FS, Robbins SL, Denton J, Nookala L, Meeker D, Ohno-MaChado L, et al. Creating a common data model for comparative effectiveness with the observational medical outcomes partnership. Appl Clin Informatics. (2015) 06:536–47. doi: 10.4338/ACI-2014-12-CR-0121

CrossRef Full Text | Google Scholar

47. Lemaître G, Nogueira F, Aridas CK. Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning. J Mach Learn Res. (2017) 18:559–63.

Google Scholar

48. Youden WJ. Index for rating diagnostic tests. Cancer. (1950) 3:32–5. doi: 10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Chan KK, Bass AR. Monitoring and management of the patient with immune checkpoint inhibitor-induced inflammatory arthritis: current perspectives. J Inflammation Res. (2022) 15:3105–18. doi: 10.2147/JIR.S282600

CrossRef Full Text | Google Scholar

50. Gómez-Puerta JA, Lobo-Prat D, Perez-García C, Ponce A, Frade-Sosa B, Millán Arciniegas AM, et al. Clinical patterns and follow-up of inflammatory arthritis and other immune-related adverse events induced by checkpoint inhibitors. A multicenter study. Front Med. (2022) 9. doi: 10.3389/fmed.2022.888377

CrossRef Full Text | Google Scholar