Frank Rojas
Sharia Hernandez
Rossana Lazcano
Caddie Laberiano-Fernandez
Edwin Roger Parra- Department of Translational Molecular Pathology, The University of Texas MD Anderson Cancer Center, Houston, TX, United States
A robust understanding of the tumor immune environment has important implications for cancer diagnosis, prognosis, research, and immunotherapy. Traditionally, immunohistochemistry (IHC) has been regarded as the standard method for detecting proteins in situ, but this technique allows for the evaluation of only one cell marker per tissue sample at a time. However, multiplexed imaging technologies enable the multiparametric analysis of a tissue section at the same time. Also, through the curation of specific antibody panels, these technologies enable researchers to study the cell subpopulations within a single immunological cell group. Thus, multiplexed imaging gives investigators the opportunity to better understand tumor cells, immune cells, and the interactions between them. In the multiplexed imaging technology workflow, once the protocol for a tumor immune micro environment study has been defined, histological slides are digitized to produce high-resolution images in which regions of interest are selected for the interrogation of simultaneously expressed immunomarkers (including those co-expressed by the same cell) by using an image analysis software and algorithm. Most currently available image analysis software packages use similar machine learning approaches in which tissue segmentation first defines the different components that make up the regions of interest and cell segmentation, then defines the different parameters, such as the nucleus and cytoplasm, that the software must utilize to segment single cells. Image analysis tools have driven dramatic evolution in the field of digital pathology over the past several decades and provided the data necessary for translational research and the discovery of new therapeutic targets. The next step in the growth of digital pathology is optimization and standardization of the different tasks in cancer research, including image analysis algorithm creation, to increase the amount of data generated and their accuracy in a short time as described herein. The aim of this review is to describe this process, including an image analysis algorithm creation for multiplex immunofluorescence analysis, as an essential part of the optimization and standardization of the different processes in cancer research, to increase the amount of data generated and their accuracy in a short time.
Introduction
The past several decades have seen growth in the use of cancer-screening protocols in the general population. These advances have increased the amount of material and work dedicated to cancer research and diagnosis. Cancer research and diagnosis use medical images such as mammography, magnetic resonance imaging, ultrasound, and microscopic tissue images (1), One of the most actively researched tasks in development is digital image analysis using computer-assisted diagnosis (2–6). Digital image analysis has become an essential part of cancer research, detection, treatment decisions, and surveillance routines, and it can be potentially used at many screening sites globally (7–10). Indeed, this technique has the much-needed potential to relieve pathologists’ workloads, diminish subjectivity, improvement of performance, and accuracy, allowing increased attention toward more challenging cases (1, 7, 10).
To this end, multiplexed imaging technologies have proven to be valuable techniques that enable the simultaneous detection of multiple markers in a single tissue section. Increasing demand for clinical trials evaluating large numbers of samples despite relatively limited amounts of tissue (especially brain tissue and other rare tumor tissue) available for research purposes has only increased the importance of these techniques (9). In cancer immunotherapy specifically, these tools have proven to be indispensable. For example, in numerous studies concerning programmed death-ligand 1 (PD-L1), multiplexed imaging technologies seemed to improve performance in predicting response to anti-PD-L1/programmed cell death protein 1 (PD-1) treatment of different solid tumors, as well as tumor mutational burden and gene expression profiling (11–13). Investigators have implemented and adopted multiplexed imaging technologies in research, and they will probably do so in clinical settings in the near future, enabling detailed cell structure, functional state, and cell-cell interaction analysis (14).
To take full advantage of multiplexed imaging technologies capabilities, such as telepathology, second opinions, education, and big data generation, cancer researchers have made many advances in histopathological whole slide image analysis (15). For example, improved digital image analysis software has enabled the compartmentalization of tissue into its components by selecting representative tissue compartment examples, such as tumor and stroma (Figure 1). In addition, lymphoid tissue and glass/background categories can be included in the tissue compartmentalization process; the former would be helpful for tissue morphometric analysis. Furthermore, detecting their expressed proteins through sophisticated algorithms generates large amounts of digital information that can be integrated with clinical information thanks to medical digitizing technologies (2, 15, 16). Histological analysis of tissue patterns by using computer-aided image processing to perform disease classification is made possible by significant developments in digitized histological studies (7–9, 15, 17). Similarly, quantitative pathological image characterization is essential for clinical applications, as it decreases interobserver and intra-observer variations and for research applications, particularly for understanding the development mechanisms and biology of cancer (7). Furthermore, quantitative analysis of immunohistochemically stained samples for immune cells or tumor biomarkers via digital pathological approaches increases the precision and accuracy in measurement of the expression of surface proteins in pretreated and treated samples to evaluate the tumor microenvironment (TME) composition and its modifications in response to immunotherapy or targeted therapy (18).
Figure 1 egion of Interest and Image Preparation for Multiplex Analysis Using Tyramide Signal Amplification. Representative digital image of non-small cell lung cancer for multiplex immunofluorescence phenotyping analysis (10x – scale bar: 500 μm). (A) Hematoxylin & Eosin (H&E) (representative H&E image view provided by the software during analysis). (B) Multiplex immunofluorescence image view for ROI (composite image). (C) Multiplex immunofluorescence image on intratumoral region with training regions for tissue segmentation, red (tumor), green (stroma), blue (glass). (D) Region on interest (intratumoral) after training algorithm for tissue segmentation, red (tumor), green (stroma), blue (glass). Composite image from inForm ® image analysis software, Akoya bioscience. Scanner Vectra Polaris.
This review aims to explain how multiplexed imaging technologies are applied to translational research and tumor immune environment characterization. To achieve research objectives by using digital image analysis, optimizing and standardizing multiple immunofluorescence validation and panel design in this way and the different steps of the image analysis workflow, which is necessary to improve imaging data accuracy.
Tumor Immune Environment
Cancer treatment has changed dramatically in the last two decades due to an increased understanding of tumor biology and its mechanisms of development, allowing the development of targeted therapy (19, 20). Immunotherapy enhances the body’s antitumor immune response by promoting tumor immune recognition, immune activation, and immune response persistence. As a result, it has become a valuable cancer treatment option for those suitable cancer patients (21–27). Since the development of immune checkpoint inhibitors (ICIs), the incidence of postoperative recurrence of cancer has decreased, and progression-free survival (PFS) and overall survival (OS) have improved (28–30). As an example, Nadim clinical trial (NCT03081689), a multicenter open-label clinical trial in which 46 patients with resectable non-small cell lung cancer (NSCLC) treated with neoadjuvant chemo-immunotherapy, were evaluated, it was reported a progression-free survival (PFS) of 36 months and overall survival (OS) of 42 months (30). Furthermore, a systematic review of the literature has also reported improvement in 12 months’ overall survival of cancer patients treated with immunotherapy, alone or in combination (31). However, although immune checkpoint inhibitors have demonstrated high efficacy in the treatment of tumors such as melanoma, non-small cell lung carcinoma, renal cell carcinoma, and Hodgkin lymphoma (32, 33), treatment-related adverse effects and toxicity are still an important issue in these patients, and more studies are needed (34, 35).
Cancer Immunoediting
Tumor differentiation, epigenetics, tumor spread, and immune evasion are all influenced by the TME. The TME is very diverse and is constituted by several cell types and a wide variety of chemical molecules produced and released by tumor cells, stromal cells, and other cells (36) (Figure 2). Both innate and adaptive immunity have roles in immune surveillance, also known as cancer immunoediting, with three phases: activation, equilibrium, and escape (37). During the elimination phase, the host protective mechanisms of the immune system detect and attack the tumor cells that express stress-related molecules. Some of these molecules include lymphocytes’ host effector molecules such as interferon-gamma and perforin, targeting tumor cells and enhancing the cross-presentation between dendritic cells (DCs) CD103+ and CD8+ T cells, among other immune phenomena (37–40). Tumor cells arise when altered cells escape immune control during this phase; despite the immune system’s ability to recognize and kill tumor cells, a tumor can continue to grow (equilibrium phase) and eventually escape surveillance (escape phase) (37, 41).
Figure 2 Tumor Microenvironment (TME). PD-L1 tumoral cell expression and PD-1 immune cell axis plays a key role in physiological immune homeostasis and contributes to tumor cell immune evasion. Cytotoxic T-lymphocyte antigen 4 (CTLA-4) ligation to CD28 quantitatively augments TCR-mediated signals, as well as activating independent pathways to upregulate CD28-mediated cytokine production and proliferation, raising the threshold needed for T-cell activation and arresting T-cell-cycle progression. Simultaneous recognition of a specific major histocompatibility complex (MHC)–peptide complex by the T-cell receptor (TCR) and of B7-1 (CD80) or B7-2 (CD86) also results in T-cell activation, cytokine production, proliferation and differentiation. Natural killer (NK) cells are effector lymphocytes that play protective roles against both infectious pathogens and cancer. Dendritic cells (DC) are professional antigen presenting cells, able to induce naïve T cell activation and effector differentiation, involved in the induction and maintenance of immune tolerance in homeostasis. Tumor associated macrophages (TAMs) are constituted by: M1 macrophages that play critical roles in innate host defense by producing reactive oxygen/nitrogen species (ROS/RNS) and pro-inflammatory cytokines such as IL-1β, IL-6, tumor necrosis factor α (TNF-α), and M2 macrophages that produce anti-inflammatory cytokines such as IL-10, IL-13 and TGF-β to promote tumor development. Image created with BioRender.com.
Antitumor and Pro-Tumor Immune Subsets
Antitumor immune responses are mediated by lymphocytes such as natural killer (NK) cells, CD8+ T cells, and CD4+ helper T (Th) cells, as well as proinflammatory (M1) macrophages and DCs. In contrast, pro-tumor immunity is mediated by a heterogeneous population of myeloid-derived suppressor cells (MDSCs) and Foxp3+ regulatory T cells (Tregs) (36). Tumor cells evade immune cells through various mechanisms, including inhibition or loss of tumor antigens, the release of immunosuppressive extracellular vesicles such as exosomes, the release of immunosuppressive molecules such as interleukin-10, and transforming growth factor (TGF). Other described mechanisms are shedding of soluble major histocompatibility complex I (MHC-I), loss of adhesion molecules such as intercellular adhesion molecule-1 (ICAM-1), development of apoptosis through tumor necrosis factor-related apoptosis-inducing ligand (TRAIL), resistance to apoptosis via upregulation of BCL-2, and induction of immune tolerance by the extensively used therapeutic target PD-L1 (36).
Dendritic Cells
DCs are critical immune system regulators that also coordinate immune responses in tumors, even though other innate immune cells also play a role in immune surveillance and immune tolerance; for example, M1 macrophages neutralize tumors by secreting lytic enzymes, tumor necrosis factor-alpha, and oxygen and nitrogen reactive intermediates and by inducing antibody-dependent cell-mediated cytotoxicity (ADCC) (42, 43).
Tumor antigens are successfully engulfed and processed by immature DCs in the TME. However, DC maturation and activation are elicited by damage-associated molecular patterns, which cause DCs to lose their ability to collect antigens and gain the ability to offer these antigens to lymphocytes. Mature DCs move from the tumor surrounding tissue to lymph nodes, where they offer antigens to CD4+ and CD8+ T lymphocytes via major histocompatibility complex class II and I molecules, respectively (36). In some cases, despite its ability to mount antitumor immunity, the immune system is unable to limit the tumor’s growth. This tumor immune evasion is just as challenging to activate as immune activation. Therefore, a lack of appropriate DC stimulation is critical in the tumor immune environment. If the DCs in the TME are not fully matured, they will present tumor antigens in a tolerogenic manner, resulting in anergic/tolerant T cells. As juvenile DCs in the TME, regulatory and tolerogenic DCs (tDCs) exhibit low expression of co-stimulatory molecules like CD80 and CD86 but high expression of inhibitory molecules like PD-L1 and cytotoxic T-lymphocyte-associated antigen-protein 4 (CTLA-4) (44).
Tumor Infiltrating Lymphocytes
Tumor-infiltrating lymphocytes are lymphocytes that infiltrate the tumor site as a result of molecular signals (41). These TILs are constituted by different T-cell subsets, such as regulatory T cells (Tregs) and innate lymphoid cells (ILCs), such as natural killer (NK) cells, among others. Effector T cells and natural killer cells are commonly elicited and attracted to the TME to kill cancer cells by recognizing tumor antigens and membrane ligands. In addition, immune cell infiltration, tumor cell growth, and metastasis are influenced by chemokines, selectins, and integrins secreted by tumor cells and their stroma, which also aid in the extravasation tumor-infiltrating lymphocytes into tissues (45).
The Traditional Pathological Tissue Analysis
Pathologists have used hematoxylin and eosin staining to visualize multiple tissue components in microscopic examinations of biopsy and surgical samples for more than 100 years (7). In histopathological analysis, disease grading and diagnosis include recognizing tissue structures such as lymphocytes, cancer nuclei, and glands. The nature, degree, size, shape, and other morphological features of these structures are essential indicators of the presence and severity of the disease (46).
In traditional pathological slide analysis, optic microscopic assessment at different magnifications (e.g., 4x, 10x, 20x, 40x, 100x) provides different levels of information. High-power field microscopy provides information about cell shape, whereas low-power field microscopy provides structural understanding and enables the identification of complex architectural features and spatial relations (47). However, in digital image analysis, the quality, accuracy, and amount of information depend on the image resolution, field of view, and selected regions of interest when performing digital image analysis. In the same way, to identify target molecules expression with clinical utility and their semiquantitative intensity assessment in the tissue, IHC has been an essential auxiliary method for pathologists not only in the diagnostic routine but also in the research field, providing a high-performance assessment tool when compared with other biomarkers modalities (11, 48–50),
In digital image analysis, pyramidal image and whole-slide analysis can address the lack of architectural data when using high-power field magnification. In addition, because cellular and structural atypia are present in cancerous tissues, images of tumor sections obtained at different magnifications contain important data (47).
For example, when grading prostate cancer, some characteristics are associated with tumor aggressiveness, such as cell features, tissue architecture, and image complexity, which are important for determining the Gleason patterns. However, for some challenging cases, these histological features can be influenced by interobserver variability, and although M2 macrophages and regulatory T cells have been suggested as part of the immune-suppressive mechanisms that contribute to cancer progression (51), tumor immune environment characteristics in prostate cancer are still under investigation (52, 53). On the other hand, in breast cancer samples and tumor immune environment, has been described the dual role of the immune system in tumor development/progression and inhibition (54–56). In tumor immune environment studies of breast cancer patients, a complex dysfunction of T helper 2 cells associated with a low percentage of CD4+ and CD8+ cells has also been reported (57, 58). Furthermore, researchers have observed that large numbers of lymphocytes are highly predictive of a good outcome and improved survival in histopathological assessment (46, 54, 56).
Another assessment for breast cancer is the study of tumor infiltrating lymphocytes (TILs) as a strong prognostic biomarker of good outcome for early-stage triple negative breast cancer (TNBC) and HER2+ breast cancer, which has been included in several international guidelines such as St. Gallen consensus (59), the European society of medical oncology (ESMO) (60), and the World Health Organization (WHO), as a criteria for pathology assessment and reporting (61–63).
Due to the high relevance of the tumor immune environment and the dynamic interrelations among its components for developing novel therapeutic strategies, a high degree of inter-observer and intra-observer variability in tissue analysis is a challenge for pathologists. So digital image analysis in translational research for cancer immune environment represents a good opportunity to address this issue and also generate big data from the whole slide image analysis by using multiplexed technologies for phenotyping (17, 56, 58, 64, 65).
Tumor Immune Microenvironment Research Approaches
In immuno-oncology research, understanding the complexity of the tumor immune microenvironment is essential. The study of the tumor immune environment is made possible by obtaining immune signature data using different techniques, such as flow cytometry, IHC, and multiplex immunofluorescence (mIF). In translational research, the panel design process requires the collaboration of a multidisciplinary team and is constituted by consecutive steps pursuing to find the answer to the scientific question (Figure 3).
Figure 3 Study and multiplex immunofluorescence panel design in translational research. Multiplex Immunofluorescence (mIF) panel design for translational research starts with a scientific question and a multidisciplinary team guided by a scientific leader who addresses their skills and efforts to accomplish the research goal. Sample collection, data review, and pathologist quality control are simultaneously applied to determine the study’s feasibility. Antibody validation is an essential part of the process and assures that the expressed surface proteins and their intensity are correctly detected. High-resolution image scanning provides high-quality images for analysis purposes. Finally, when the image analysis algorithm is applied, data results are exported, analyzed, and correlated. Image created with BioRender.com.
Although flow cytometry has been used for immunophenotyping, this method requires cell disaggregation, which leads to loss of tissue architecture and information on spatial relationships between tumor and immune cells (66). IHC has also been used for immune cell phenotyping, and this technique preserves tissue architecture and the spatial relationship among cells. However, the amount of molecular proteins that can be detected simultaneously is limited (11, 50, 67). In the same way, the analysis of multiplexed images preserves the tissue architecture and the spatial relationships among all of the cell populations, which makes it a handy tool for characterizing the TME in translational research and obtaining reliable information (68–70). However, each technique offers a different perspective and data, so flow cytometry, IHC, and mIF must be considered complementary techniques and not excluded.
In addition, mIF can be used to evaluate protein localization in tissue samples using an immunofluorescent labeling-based image analysis algorithm (68). Also, given a reasonable number of markers in the same tissue section, it can facilitate the exploration of tissue morphology, spatial distributions, subcellular features, and cell surface proteins (71). Different technologies are capable of capturing the immune signatures of cells; in mIF, once different fluorochromes are attached to tissue proteins, signals can be extracted according to wavelength band from a series of scanned images, and the signals from various fluorophores are separated according to the signals created by Opal dyes (72, 73). To produce enzymatic signal amplification, tyramide molecules are conjugated with these fluorescent dyes (74).
An essential component of computerized image analysis in digital pathology is ascertaining prognostic markers’ function and expression patterns to predict cancer outcomes and survival (17, 64, 65). In this sense, immunotherapies have provided an increased understanding of patients’ TME and clinical response to treatment with immunotherapy. In addition, they have enabled clinicians to determine the appropriate indications for cancer treatment and identify possible effective drug combinations. For example, despite the efficacy of inhibitors of programmed cell death protein 1 (nivolumab and pembrolizumab) and PD-L1 (atezolizumab and durvalumab) in the treatment of melanoma and other solid tumors, physicians have observed resistance and relapse in other cancer patients with PD-L1 expression in malignant cells (75, 76). Similarly, investigators identified the simultaneous presence of cytotoxic CD8+ cells and expression of PD-L1 in tumors as a reliable prognostic marker for gastric cancer (77). From these findings, we can infer that due to the complexity and diversity of the tumor immune environment, more sophisticated techniques are required to identify and study different immune cell subsets and analyze cell-cell interactions in human tumor samples (78).
Multiplex Immunoflourescence for Immune Cell Profiling
Several technical considerations must be addressed for developing a consistent mIF imaging platform, such as comprehensive tissue quality standards, a standardized multiplex assay staining scheme, the ability to quantify several markers in a specific region of interest, and experimental reproducibility both within and between laboratories (79).
Antibodies for Multiplex Immunofluorescence
Because of their superior specificity, reliability, low frequency of staining background, and lot-to-lot variability, monoclonal antibodies are frequently used for immunohistochemical and immunofluorescent panel validation and analysis. Also, when compared to polyclonal antibodies, monoclonal antibodies are specific to target antigens, whereas polyclonal antibodies bind to various epitopes on the same protein and are acquired from experimental animals via repeated antigen stimulation (80).
Tyramide Signal Amplification and Target Biomarkers
mIF tyramide signal amplification (TSA) combines staining with multispectral imaging analysis and allows for the creation of mIF panels of up to eight biomarkers, quantifies their expression, and characterizes their co-expression (cell phenotypes). A combination of biomarkers and individual fluorophores creates multispectral images that can generate several mIF panels for studying the tissue microenvironment (14, 80) (Figures 4, 5). Targetable biomarkers like programmed cell death protein 1 PD-1/PD-L1 and their pathways can be analyzed to confirm the effect of immune treatments on the TME and their therapeutic benefit. Tumor immune environment analysis by mIF has been used to predict prognosis for diseases other than cancer and in the early phases of pathogenesis when signaling protein levels and functions are disrupted. Therefore, this technology (mIF) has played an important role in translational oncology research by increasing the understanding of the natural progress of the disease (81, 82).
Figure 4 Multiplex immunofluorescence activation panel. The activation panel includes a universal biomarker for nuclear detection (DAPI) and cytokeratin for the detection of epithelial cells (malignant cells), which is also helpful for the compartmentalization of the image (tumor vs. stroma). TME and its different immune cell populations are detected using CD3+ for all lymphocytes and co-expressed CD8+ for cytotoxic T cells. Biomarker CD68+ identify macrophage population. This panel also includes biomarker expression for PD1+ and PDL1+ in immune cells and tumor cells to determine their activator or inhibitor status. (A) Composite image of TME from oral squamous cell carcinoma, showing all markers of a multiplex immunofluorescence “activation panel”, activated simultaneously. (B) Composite image showing CD68 positive expression (yellow) of immune cells in the stromal compartment. (C) Composite image showing PD1 positive expression (green) of immune cells in the stromal compartment. (D) Composite image showing CD3 positive expression (red) of immune cells in the stromal compartment. (E) Composite image showing CD8 positive expression (pink) of immune cells in the stromal compartment. (F) Composite image showing PD-L1 positive expression (orange) of immune cells in the stromal compartment. inForm ® image analysis software, Akoya bioscience (scale bar: 100 μm). Scanner Vectra Polaris. DAPI (blue-DAPI), cytokeratin (cyan-opal 620), CD3+ (red-opal 690), CD8+ (pink-opal 540), CD68+ (yellow-opal 520), PD1+ (green-opal 650), PD-L1+ (orange-opal 570).
Figure 5 Multiplex immunofluorescence T lymphocytes panel. As on the activation panel, the T lymphocytes panel includes a universal biomarker for nuclear detection (DAPI) and cytokeratin for detection of epithelial cells (malignant cells), which is also helpful for compartmentalization of the image (tumor vs. stroma). T lymphocyte populations are detected using CD3+ for all lymphocytes, co-expressed CD8+ for cytotoxic T cells, and co-expressed FOXP3+ for regulatory T-cells. This panel includes granzyme B+ (GB) and CD45RO+ to identify activated T-cells and memory T-cells, respectively. (A) Composite image of TME from oral squamous cell carcinoma, showing all markers of a multiplex immunofluorescence “T lymphocytes phenotypes”, activated simultaneously. (B) Composite image showing CD45RO positive expression (yellow) of immune cells in the stromal compartment. (C) Composite image showing FOXP3 positive expression (green) of immune cells in the stromal compartment. (D) Composite image showing CD3 positive expression (red) of immune cells in the stromal compartment. (E) Composite image showing CD8 positive expression (pink) of immune cells in the stromal compartment. (F) Composite image showing GB positive expression (magenta) of immune cells in the stromal compartment. inForm ® image analysis software, Akoya bioscience (scale bar: 100 μm). Scanner Vectra Polaris. DAPI (blue-DAPI), cytokeratin (cyan-opal 620), CD3+ (red-opal 690), CD8+ (pink-opal 540), CD45RO+ (yellow-opal 520), FOXP3+ (green-opal 650), Granzyme B+ (magenta-opal 570).
Individual markers, such as those found in mIF panels, can be expressed in several types of cells simultaneously, and the detection threshold of the stained subcellular compartments, such as nuclei, cytoplasm, and cell membranes, must be addressed when creating the analysis algorithm. For example, tumor cells, macrophages, and lymphocytes can express PD-L1, but because lymphocytes are small cells with little cytoplasm that cannot always be differentiated from the cell membrane, they should be considered as positive for PD-L1 expression when cytoplasmic or membrane expression is present (83).
Markers Co-Expression
Identifying the co-expression of several markers in specific cells is a strategy for confirming the specific cell target protein by including negative controls during the validation of the panel markers. In addition, including negative controls help eliminate autofluorescence interference during image preparation to ensure the acquisition of a clean signal for TME and mIF analysis (83).
For different mIF biomarker panels, including a common marker in the panels as an internal control is critical. For example, in immuno-oncology research, CD3 is commonly employed to analyze lymphocytes and their different subpopulations. In addition, although various tissue cut levels of formalin-fixed paraffin-embedded (FFPE) biopsy samples are utilized in the staining process, consecutive tissue cut levels should be used to obtain similar cellularity among panels during the staining procedure (83).
Cell phenotypes can be detected in any tissue compartment. They are defined by the co-expression of two or more surface markers simultaneously expressed in the known cell compartment during image analysis (83, 84) (Table 1).
Table 1 Examples of immune cells phenotypes according to their positive surface marker expression used in translational research.
Tissue Processing and Effects on Multiplexed Images
Tissue processing and fixation are critical steps that must be standardized to avoid pre-analytical and analytical errors. Tissue ischemia and inappropriate tissue fixing can cause autolysis and degradation of proteins, RNA, DNA, and enzymes, decreasing antigenicity and antibody-binding site. Digital image analysis data from mIF can be negatively affected if extensive areas of necrosis, hemorrhage, and fatty tissue are present or a previous decalcification method was used (85). Some features must be considered regarding the tissue area selection, such as sample size, discordant diagnosis, amount of tumor cells, and selection of immune hot versus immune cold areas, among others (15, 85).
Image Analysis Algorithm Creation
General Perspectives
Tissue analysis of oncological pathologies by optic microscopy is made by manipulating the microscope’s magnification lens to gather information at the architectural and cellular levels. In machine learning-based digital image analysis, different images obtained at various magnification and quality resolution levels are often used (47, 86). Examination of whole slide images is the best approach to digital image analysis in histopathology because it yields reliable and accurate data and comprehensive information on spatial relationships among the different surface proteins expressed in the TME (87). However, since the amount of data from whole slide image analysis can be extensive and data analysis time consuming, if the experimental design allows, tissue microarray (TMA) can be used as an alternative technique that allows high throughput analysis of multiple tissue specimens at the same time, under experimental uniformity, amplifying scarce resources and decreasing volume assay, time and cost (88, 89).
Noteworthy, TME and tissue heterogeneity may vary throughout the whole tissue specimen. Therefore, the selected areas for analysis could be affected during the selection, so several studies have tried to apply AI algorithms under pathologist supervision to identify diagnostically important regions of interest (ROIs). These studies used image features such as pixels and textures, achieving tissue classification accuracy between 46% and 75% based on the quality of the tissue, quality of the image, and training data (90–92). These techniques must be selected according to the study objectives, and they should be considered complementary.
Many image analysis software packages are commercially available (Table 2), and they can be used for the phenotyping of samples through the input of specific tissue and cell features. In addition, open-source digital image analysis software packages are available, such as ImageJ (National Institutes of Health) and CellProfiler (Table 3). The analysis workflow of histopathological digital images carried out by using the different and currently available software packages are pretty similar. They all use reference points to recognize different structures that allow the software for unsupervised image exploration, image classification, image segmentation, and object tracking to identify surface proteins in mIF technology expressed in various tissue compartments (93).
Table 2 Representative and commercially available digital image analysis software packages.
Table 3 Representative open source digital image analysis software packages.
In developing an image analysis algorithm, machine learning and deep learning methods for hierarchal data clustering can be obtained by using the k-means method, which is a popular deep learning method, and density-based spatial clustering of application with noise (DBSCAN), another deep learning method that differs from the K-means on its less susceptibility for extremes median values (93–96).
The intensity of marker expression and morphology is essential for tissue classification, cell classification, and identification and measurement of protein expression. Then, the tumor sample images are analyzed using the principle of mutual identification and exclusion once the tissue characteristics are determined based on the immunological background regarding the tumor’s behavior reported by scientific literature and its relationship with the immune system according to the study’s objectives (97). Finally, a binary classification scheme is used to classify cells as positive or negative for a single marker. In this classification, the assessment of the surface markers is based on a cell attribute that is not shared by the rest of the cell population (78, 98).
When evaluating a large number of digital images, a combination of deep learning algorithms and machine learning is helpful in the analysis of mIF data and quantification of image features (97). For these algorithms to work effectively, they must recognize, learn from, and dynamically adjust to tissue heterogeneity. However, an algorithm may perform well for one image but poorly for another using fixed input parameters (99), making extracting information from digital slide images and analyzing it challenging. Once the images have been prepared by activating the fluorochrome detection for multiplexed digital image analysis, the following steps consist of tissue segmentation, cell segmentation, phenotyping, and data exportation (Figure 6).
Figure 6 Algorithm Workflow of Multiplex immunofluorescence Digital Image Analysis. After image scanning, raw images (A) are prepared by activating the fluorochromes attached to the different cell surface proteins (B). Then, for tissue compartmentalization, the software user can select the most representative examples of each compartment used during the training algorithm and exclude areas of disinterest (C). Once the tissue compartment algorithm is applied and the areas are defined (D), the next step is to define the cells limits (E) as individual objects to count for the analysis. The final part of the image analysis algorithm is the classification of the cells based on their phenotypes according to their surface protein expression (F). Images examples from inForm ® image analysis software. Akoya Biosciences.
Tissue Segmentation
Image compartmentalization is one of the first steps in digital image analysis and algorithm creation. It demarcates a tissue sample’s compartments, such as the tumor, stroma, blood vessels, and other histomorphology areas. During this step, the software user must create and name the different tissue categories of interest and provide representative examples of them to achieve an accurate detection of the different tissue categories provided by the expert (Figure 1). The number of examples needed for each category may vary depending on the tissue heterogeneity and tissue preparation quality, staining quality, and image resolution. Usually, providing some of the best representative examples from the training areas is considered good practice, but the overall accuracy of tissue classification always depends on the quality of the training areas selected by the user and the software’s capabilities in interpreting those areas.
The user can also separate specific areas of disinterest, such as necrosis, tissue-processing artifacts, and staining artifacts. These areas can then be eliminated using the various methods embedded within a digital image analysis software program, such as manually drawing a perimeter around the object of disinterest or altering thresholds for lower or higher marker values. After doing so, the algorithm will disregard potentially confusing elements during its training process that may later put the algorithm’s accuracy at risk.
To quantify the total area of tissue to be analyzed, the user can create a new category representing the image areas that do not contain tissue. Some suggested names for this category are “glass” and “background” (94, 100). This helps the software save time when performing image analysis, as it can be configured during training to not apply the final image analysis algorithm to the areas without tissue.
Cell Segmentation
Cell segmentation begins with nuclear detection. The digital image analysis algorithm combines different nuclear characteristics, the most important of which include size, roundness, edges, and texture, to identify the individual nuclei in a sample. Some tumor cells with high nuclear density may exhibit nuclear overlap, which may also be seen in neoplastic samples with high-density immune cell populations. However, this characteristic may be erroneously overlooked as a tissue artifact in either case. To address this nuclear overlapping issue, most digital image analysis software packages permit the regulation of nuclear segmentation aggressiveness. This function enables the splitting of a nuclear signal into the areas where the software user can identify more than one nucleus and thus prevents underestimation or overestimation of the cell population.
The cytoplasm and membrane are also considered critical features in each cell during cell segmentation, depending on the digital image analysis software program and its proper use.
Although a counterstaining nuclear marker (e.g., DAPI) should always be included in an mIF staining panel, a marker like cytokeratin, which is used for the identification of epithelial tumors, also can be employed as a cytoplasmic/membrane marker to improve segmentation of cells (85). In addition, the segmentation of immune cells also can be achieved by using membrane markers such as CD68 for macrophages and CD3 for T cells.
Phenotyping, Single-Cell Marker Expression, and Cell Marker Co-Expression
An mIF marker panel can be analyzed using single-cell marker identification or according to the co-expression of two or more markers. The choice between these two approaches depends on the complexity of the markers in the panel, and the markers can be used as comprehensive or restrictive cell phenotype identifiers. In the single-cell marker approach, individual cell marker identification begins with creating an algorithm that identifies mutually exclusive markers in mIF panels, such as cytokeratin, CD3, and CD68. Then, the other markers are identified in consecutive steps in different phenotyping sessions. In the co-expression approach, co-expressed markers are identified by activating at least two different phenotypes simultaneously (Figure 7).
Figure 7 Multiplex immunofluorescence and colocalization examples. (A) CD3+ expression (red) on a lymphocyte T. (B) CD8+ expression (pink) of the same lymphocyte T in example (A). (C) CD3+ and CD8+ expression and colocalization (red and pink) for a cytotoxic T lymphocyte. (D) CD3+ expression (red) on a lymphocyte T. (E) PD1+ expression (green) of the same lymphocyte T in example (D). (F) CD3+ and PD1+ expression and colocalization (red and green) for a T lymphocyte PD1+. (G) Cytokeratin+ expression (cyan) of an epithelial tumor cell. (H) PDL1+ expression (orange) of the same epithelial tumor cell on example (G). (I) Cytokeratin+ and PDL1+ expression and colocalization (cyan and orange) on an epithelial tumor cell expressing PDL1 on its surface for immune evasion. Composite image from inform ® image analysis software, Akoya bioscience. Scanner Vectra Polaris.
Using a visual approach, a pathologist or analyst familiar with the specific marker’s expression and location in the cells must correctly identify the marker or cell phenotype using the algorithm and avoid staining of the background and false positivity (85, 101). The co-expression of markers is accomplished by activating two or three markers simultaneously and verifying that they overlap in the correct pattern in the expected cells. However, using a control marker’s intensity when possible and adequate for the performance and training of the algorithm is recommended.
Digital image analysis for multiplex immunofluorescence can be performed in single or batch mode, both of which have advantages and disadvantages. Batch mode is faster and enables the analysis of a large group of samples in a shorter time than single-mode. However, due to the heterogeneity of the intrinsic tumor tissue and technical differences due to tissue processing, such as fixation and staining, applying a single algorithm in batch mode may be challenging, sometimes misreading tissue compartments and under/over detecting single nuclei. In addition, the application of a single digital image analysis algorithm requires selecting a training set group of images, which should be constituted by some of the most representative samples included in the study and show special characteristics that might present, such as mucin, cartilage, hemosiderin-laden macrophages, and others. Finally, to address the lack of accuracy during tissue and cell classification, the ideal software should allow real-time quality control supervised by a pathologist; whereas single-mode analysis can be more accurate than batch mode analysis, the former takes more time.
In addition, a training algorithm is applied to different images, and a pathologist must verify the accuracy of that application. Quality control in machine learning-based examination of biopsy samples, determination of the sample dimensions, and verification of accurate cell populations by a well-trained pathologist help ensure high-quality data from multiplex image analysis (85). Deep learning methods generate a significant amount of relational and non-relational data. This big-scale information requires automated or semi-automated storage, processing, and analysis, offering adequate and complete information that can be filtered and improved by automation and supervised under medical expert knowledge and allowing for finding data trends on a large scale to provide decision-making support (102, 103). However, handling all the generated data can be highly costly due to the different knowledge filed involved in the process, such as medical experts, technicians, data analysts, and technology development engineers, so much improvement must be done to expand the number of images and shorten their analysis results in a costless and timely manner.
Translational Research, Digital Pathology Challenges, and Future Steps
Counting cells, blood vessels, and nuclei is a standard and necessary task in translational cancer research that can also be challenging due to interobserver variability. Thus, a primary objective of artificial intelligence-based tissue analysis is the promotion of reliability of the data obtained. For example, in comparing automated and manual object counts, authors have reported interobserver variations, finding an error range of 1.1-4.7% in the automated number of cell counts compared with manual counts (104). Similarly, in comparing manual and automated cell counts of immunohistochemically stained bone marrow biopsy samples, researchers observed a 2.8% to 10.0% discrepancy in these cell counts (105). Furthermore, the authors reported an error range of 19-42% between manual and automated mitosis counts in breast tissue samples visualized with Feulgen stain (106). These findings demonstrate that tissue analysis by pathologists can be subjective and prone to inter-observer and intra-observer variations (107). Therefore, improving the performance of object counts by using artificial intelligence is important to obtain more accurate and reproducible data on tissue samples.
Another challenge in mIF and digital image analysis is using different analytical approaches that are subject to potential errors that may affect the data output. For example, the selection of regions of interest in platforms such as mIF may be subjective, affecting the quality of the image analysis results. Therefore, whole slide images should be analyzed when possible to reduce variability in data results. Standardization of the different steps involved in any mIF project design, from panel antibody validation to the image analysis workflow, is necessary to overcome mIF and digital image analysis variations among researchers (15, 85).
The pathology community has been debating different assays to validate digital pathology in translational research, with little agreement on the parameters that should be used or how they could be used. We consider tissue quality, antibody validation, panel design, image resolution, and algorithm creation among the essential parameters for immuno-oncology and digital image analysis validation and the standardization of the analysis workflow to obtain reliable and reproducible results data (Figure 8).
Figure 8 Multiplex Immunofluorescence. Image Analysis Overview. (A) Tumor and Stroma H&E image before preparation with multiplex fluorochromes. (B) Composite image, before algorithm analysis. (C) Tissue compartmentalization, tumor (red), stroma (green). (D) Nuclear detection and nuclear segmentation in both compartments (tumor and stroma). (E) Phenotyping map. (F) Complete algorithm visualization, including tissue segmentation, nuclear detection, cell segmentation and phenotyping. Composite image from inForm ® image analysis software, Akoya bioscience (scale bar: 100 μm). Scanner Vectra Polaris.
Many new platforms, such as multiplexed ion beam imaging (MIBI), cytometry by time-of-flight, and CO-Detection by indEXing, are currently being used to perform high-plex immunoprofiling and cell phenotyping. However, all these platforms have in common the use of image analysis algorithms to manage large quantities of data. Thus, in the very near future, more studies will be needed for digital pathology validation and the establishment of a consensus regarding the assessment of results from all these platforms and the ones to come.
Conclusions
Understanding the tumor immune microenvironment is important for cancer diagnosis and treatment. Thanks to the latest technologies in histopathology and digital image analysis, the development of tools such as mIF that allow for the focused study of proteins and cell surface markers that may be considered therapeutic targets is possible. In addition, significant advances in the acquisition, storage, and processing of digital images have enabled detailed studies of large quantities of material in a small amount of time, thus guaranteeing the reproducibility of results and their application to the development of new cancer treatments. Therefore, the design, validation, and standardization of an mIF marker panel offer an answer about the TME in the translational immuno-oncology field, and the creation of algorithms for the digital analysis of histological images with the help of available software packages are fundamental parts of understanding the tumor biology and immunology. Furthermore, applying these technologies logically and systematically facilitates the processing of large samples and images that can generate valuable and helpful information for developing new immunotherapies for cancer.
Author Contributions
FR and EP conceived the idea and theme developed in this article and wrote most of the manuscript. SH, RL, and CL contributed their expertise to digital image analysis and immune profiling. EP developed the mIF technology in our laboratory and edited the manuscript. All authors approved the final manuscript.
Funding
This manuscript was supported in part by the scientific and financial backing of the Cancer Immune Monitoring and Analysis Centers and Cancer Immunologic Data Commons Network provided through NCI Cooperative Agreement U24CA224285 for the MD Anderson Cancer Immune Monitoring and Analysis Centers and Translational Molecular Pathology Immunoprofiling Laboratory, the NIH/NCI under award number P30CA016672 and used the Institutional Tissue Bank, NIH SPORE grant 5P50CA070907-18, and Cancer Prevention & Research Institute of Texas grant MIRA-RP160688.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Acknowledgments
We thank the members of the Multiplex Immunofluorescence and Image Analysis Laboratory in the Department of Translational Molecular Pathology at The University of Texas MD Anderson Cancer Center , who contribute daily to quality multiplex immunofluorescence. We thank the pathology team in the Department of Translational Molecular Pathology for the image analysis. Editorial support was provided by Mr. Don Norwood from the Research Medical Library at The University of Texas MD Anderson Cancer Center.
References
1. El Houby EMF. Framework of Computer Aided Diagnosis Systems for Cancer Classification Based on Medical Images. J Med Syst (2018) 42(8):157. doi: 10.1007/s10916-018-1010-x
2. Cancer Genome Atlas Research N, Weinstein JN, Collisson EA, Mills GB, Shaw KR, Ozenberger BA, et al. The Cancer Genome Atlas Pan-Cancer Analysis Project. Nat Genet (2013) 45(10):1113–20. doi: 10.1038/ng.2764
3. Kaushal C, Bhat S, Koundal D, Singla A. Recent Trends in Computer Assisted Diagnosis (CAD) System for Breast Cancer Diagnosis Using Histopathological Images. IRBM (2019) 40(4):211–27. doi: 10.1016/j.irbm.2019.06.001
4. Win KY, Choomchuay S, Hamamoto K, Raveesunthornkiat M, Rangsirattanakul L, Pongsawat S. Computer Aided Diagnosis System for Detection of Cancer Cells on Cytological Pleural Effusion Images. BioMed Res Int (2018) 2018:6456724. doi: 10.1155/2018/6456724
5. Shehata M, Alksas A, Abouelkheir RT, Elmahdy A, Shaffie A, Soliman A, et al. A Comprehensive Computer-Assisted Diagnosis System for Early Assessment of Renal Cancer Tumors. Sensors (2021) 21(14):4928. doi: 10.3390/s21144928
6. Mar VJ, Scolyer RA, Long GV. Computer-Assisted Diagnosis for Skin Cancer: Have We Been Outsmarted? Lancet (2017) 389(10083):1962–4. doi: 10.1016/S0140-6736(17)31285-0
7. Gurcan MN, Boucheron LE, Can A, Madabhushi A, Rajpoot NM, Yener B. Histopathological Image Analysis: A Review. IEEE Rev BioMed Eng (2009) 2:147–71. doi: 10.1109/RBME.2009.2034865
8. Mendez AJ, Tahoces PG, Lado MJ, Souto M, Vidal JJ. Computer-Aided Diagnosis: Automatic Detection of Malignant Masses in Digitized Mammograms. Med Phys (1998) 25(6):957–64. doi: 10.1118/1.598274
9. Tang J, Rangayyan RM, Xu J, El Naqa I, Yang Y. Computer-Aided Detection and Diagnosis of Breast Cancer With Mammography: Recent Advances. IEEE Trans Inf Technol Biomed (2009) 13(2):236–51. doi: 10.1109/TITB.2008.2009441
10. Cicerone MT, Camp CH. 21 - Potential Roles for Spectroscopic Coherent Raman Imaging for Histopathology and Biomedicine. In: Alfano RR, Shi L, editors. Neurophotonics and Biomedical Spectroscopy. Elsevier (2019). p. 547–70.
11. Lu S, Stein JE, Rimm DL, Wang DW, Bell JM, Johnson DB, et al. Comparison of Biomarker Modalities for Predicting Response to PD-1/PD-L1 Checkpoint Blockade: A Systematic Review and Meta-Analysis. JAMA Oncol (2019) 5(8):1195–204. doi: 10.1001/jamaoncol.2019.1549
12. Tan WCC, Nerurkar SN, Cai HY, Ng HHM, Wu D, Wee YTF, et al. Overview of Multiplex Immunohistochemistry/Immunofluorescence Techniques in the Era of Cancer Immunotherapy. Cancer Commun (Lond). (2020) 40(4):135–53. doi: 10.1002/cac2.12023
13. Cascone T, Sepesi B, Lin HY, Kalhor N, Parra ER, Jiang M, et al. A Phase I/II Study of Neoadjuvant Cisplatin, Docetaxel, and Nintedanib for Resectable Non-Small Cell Lung Cancer. Clin Cancer Res (2020) 26(14):3525–36. doi: 10.1158/1078-0432.CCR-19-4180
14. Parra ER, Francisco-Cruz A, Wistuba II. State-Of-the-Art of Profiling Immune Contexture in the Era of Multiplexed Staining and Digital Analysis to Study Paraffin Tumor Tissues. Cancers (Basel) (2019) 11(2):247. doi: 10.3390/cancers11020247
15. Aeffner F, Zarella MD, Buchbinder N, Bui MM, Goodman MR, Hartman DJ, et al. Introduction to Digital Image Analysis in Whole-Slide Imaging: A White Paper From the Digital Pathology Association. J Pathol Informatics (2019) 10(1):9. doi: 10.4103/jpi.jpi_82_18
16. Parra ER, Jiang M, Machado-Rugolo J, Yaegashi LB, Prieto T, Farhat C, et al. Variants in Epithelial-Mesenchymal Transition and Immune Checkpoint Genes Are Associated With Immune Cell Profiles and Predict Survival in Non-Small Cell Lung Cancer. Arch Pathol Lab Med (2020) 144(10):1234–44. doi: 10.5858/arpa.2019-0419-OA
17. Madabhushi A, Lee G. Image Analysis and Machine Learning in Digital Pathology: Challenges and Opportunities. Med Image Anal (2016) 33:170–5. doi: 10.1016/j.media.2016.06.037
18. Wang D, Foran DJ, Ren J, Zhong H, Kim IY, Qi X. Exploring Automatic Prostate Histopathology Image Gleason Grading via Local Structure Modeling. Annu Int Conf IEEE Eng Med Biol Soc (2015) 2015:2649–52. doi: 10.1109/EMBC.2015.7318936
19. Corrales L, Scilla K, Caglevic C, Miller K, Oliveira J, Rolfo C. Immunotherapy in Lung Cancer: A New Age in Cancer Treatment. Adv Exp Med Biol (2018) 995:65–95. doi: 10.1007/978-3-030-02505-2_3
20. Lee YT, Tan YJ, Oon CE. Molecular Targeted Therapy: Treating Cancer With Specificity. Eur J Pharmacol (2018) 834:188–96. doi: 10.1016/j.ejphar.2018.07.034
21. Xie H, Shi X, Wang G. Neoadjuvant Immunotherapy for Resectable non-Small Cell Lung Cancer. Am J Cancer Res (2021) 11(6):2521–36.
22. Ansell SM. Fundamentals of Immunology for Understanding Immunotherapy for Lymphoma. Blood Adv (2020) 4(22):5863–7. doi: 10.1182/bloodadvances.2020002537
23. Wojas-Krawczyk K, Kalinka E, Grenda A, Krawczyk P, Milanowski J. Beyond PD-L1 Markers for Lung Cancer Immunotherapy. Int J Mol Sci (2019) 20(8):1915. doi: 10.3390/ijms20081915
24. Brahmer JR, Govindan R, Anders RA, Antonia SJ, Sagorsky S, Davies MJ, et al. The Society for Immunotherapy of Cancer Consensus Statement on Immunotherapy for the Treatment of non-Small Cell Lung Cancer (NSCLC). J ImmunoTher Cancer (2018) 6(1):75. doi: 10.1186/s40425-018-0382-2
25. Queirolo P, Boutros A, Tanda E, Spagnolo F, Quaglino P. Immune-Checkpoint Inhibitors for the Treatment of Metastatic Melanoma: A Model of Cancer Immunotherapy. Semin Cancer Biol (2019) 59:290–7. doi: 10.1016/j.semcancer.2019.08.001
26. Barzaman K, Moradi-Kalbolandi S, Hosseinzadeh A, Kazemi MH, Khorramdelazad H, Safari E, et al. Breast Cancer Immunotherapy: Current and Novel Approaches. Int Immunopharmacol (2021) 98:107886. doi: 10.1016/j.intimp.2021.107886
27. Heidegger I, Pircher A, Pichler R. Targeting the Tumor Microenvironment in Renal Cell Cancer Biology and Therapy Vol. 9. Frontiers in Oncology (2019).
28. Zhong WZ, Chen KN, Chen C, Gu CD, Wang J, Yang XN, et al. Erlotinib Versus Gemcitabine Plus Cisplatin as Neoadjuvant Treatment of Stage IIIA-N2 EGFR-Mutant Non-Small-Cell Lung Cancer (EMERGING-CTONG 1103): A Randomized Phase II Study. J Clin Oncol (2019) 37(25):2235–45. doi: 10.1200/JCO.19.00075
29. Hellmann MD, Paz-Ares L, Bernabe Caro R, Zurawski B, Kim SW, Carcereny Costa E, et al. Nivolumab Plus Ipilimumab in Advanced Non-Small-Cell Lung Cancer. N Engl J Med (2019) 381(21):2020–31. doi: 10.1056/NEJMoa1910231
30. Provencio M, Nadal E, Insa A, Campelo MRG, Pereiro D, Domine M, et al. OA20.01 Long Term Survival in Operable Stage Iiia Nsclc Patients Treated With Neoadjuvant Nivolumab Plus Chemotherapy - Nadim Study. J Thorac Oncol (2021) 16(10, Supplement):S883. doi: 10.1016/j.jtho.2021.08.101
31. Kok P-S, Cho D, Yoon W-H, Ritchie G, Marschner I, Lord S, et al. Validation of Progression-Free Survival Rate at 6 Months and Objective Response for Estimating Overall Survival in Immune Checkpoint Inhibitor Trials: A Systematic Review and Meta-Analysis. JAMA Network Open (2020) 3(9):e2011809. doi: 10.1001/jamanetworkopen.2020.11809
32. Ribas A, Wolchok JD. Cancer Immunotherapy Using Checkpoint Blockade. Science (2018) 359(6382):1350–5. doi: 10.1126/science.aar4060
33. O'Donnell JS, Hoefsmit EP, Smyth MJ, Blank CU, Teng MWL. The Promise of Neoadjuvant Immunotherapy and Surgery for Cancer Treatment. Clin Cancer Res (2019) 25(19):5743–51. doi: 10.1158/1078-0432.CCR-18-2641
34. El Majzoub I, Qdaisat A, Thein KZ, Win MA, Han MM, Jacobson K, et al. Adverse Effects of Immune Checkpoint Therapy in Cancer Patients Visiting the Emergency Department of a Comprehensive Cancer Center. Ann Emergency Med (2019) 73(1):79–87. doi: 10.1016/j.annemergmed.2018.04.019
35. Song W, Shen L, Wang Y, Liu Q, Goodwin TJ, Li J, et al. Synergistic and Low Adverse Effect Cancer Immunotherapy by Immunogenic Chemotherapy and Locally Expressed PD-L1 Trap. Nat Commun (2018) 9(1):2237. doi: 10.1038/s41467-018-04605-x
36. Labani-Motlagh A, Ashja-Mahdavi M, Loskog A. The Tumor Microenvironment: A Milieu Hindering and Obstructing Antitumor Immune Responses. Front Immunol (2020) 11:940. doi: 10.3389/fimmu.2020.00940
37. Mittal D, Gubin MM, Schreiber RD, Smyth MJ. New Insights Into Cancer Immunoediting and its Three Component Phases–Elimination, Equilibrium and Escape. Curr Opin Immunol (2014) 27:16–25. doi: 10.1016/j.coi.2014.01.004
38. Alspach E, Lussier DM, Schreiber RD. Interferon Gamma and Its Important Roles in Promoting and Inhibiting Spontaneous and Therapeutic Cancer Immunity. Cold Spring Harb Perspect Biol (2019) 11(3). doi: 10.1101/cshperspect.a028480
39. Paludan SR, Pradeu T, Masters SL, Mogensen TH. Constitutive Immune Mechanisms: Mediators of Host Defence and Immune Regulation. Nat Rev Immunol (2021) 21(3):137–50. doi: 10.1038/s41577-020-0391-5
40. Bates JP, Derakhshandeh R, Jones L, Webb TJ. Mechanisms of Immune Evasion in Breast Cancer. BMC Cancer (2018) 18(1):556. doi: 10.1186/s12885-018-4441-3
41. Swann JB, Smyth MJ. Immune Surveillance of Tumors. J Clin Invest (2007) 117(5):1137–46. doi: 10.1172/JCI31405
42. Gul N, Babes L, Siegmund K, Korthouwer R, Bogels M, Braster R, et al. Macrophages Eliminate Circulating Tumor Cells After Monoclonal Antibody Therapy. J Clin Invest (2014) 124(2):812–23. doi: 10.1172/JCI66776
43. Urban JL, Shepard HM, Rothstein JL, Sugarman BJ, Schreiber H. Tumor Necrosis Factor: A Potent Effector Molecule for Tumor Cell Killing by Activated Macrophages. Proc Natl Acad Sci U S A (1986) 83(14):5233–7. doi: 10.1073/pnas.83.14.5233
44. Gregori S, Tomasoni D, Pacciani V, Scirpoli M, Battaglia M, Magnani CF, et al. Differentiation of Type 1 T Regulatory Cells (Tr1) by Tolerogenic DC-10 Requires the IL-10-Dependent ILT4/HLA-G Pathway. Blood (2010) 116(6):935–44. doi: 10.1182/blood-2009-07-234872
45. Ben-Baruch A. Organ Selectivity in Metastasis: Regulation by Chemokines and Their Receptors. Clin Exp Metastasis (2008) 25(4):345–56. doi: 10.1007/s10585-007-9097-3
46. Alexe G, Dalgin GS, Scanfeld D, Tamayo P, Mesirov JP, DeLisi C, et al. High Expression of Lymphocyte-Associated Genes in Node-Negative HER2+ Breast Cancers Correlates With Lower Recurrence Rates. Cancer Res (2007) 67(22):10669–76. doi: 10.1158/0008-5472.CAN-07-0539
47. Song Y, Zhang L, Chen S, Ni D, Lei B, Wang T. Accurate Segmentation of Cervical Cytoplasm and Nuclei Based on Multiscale Convolutional Network and Graph Partitioning. IEEE Trans BioMed Eng (2015) 62(10):2421–33. doi: 10.1109/TBME.2015.2430895
48. Kim SW, Roh J, Park CS. Immunohistochemistry for Pathologists: Protocols, Pitfalls, and Tips. J Pathol Transl Med (2016) 50(6):411–8. doi: 10.4132/jptm.2016.08.08
49. Taube JM, Akturk G, Angelo M, Engle EL, Gnjatic S, Greenbaum S, et al. The Society for Immunotherapy of Cancer Statement on Best Practices for Multiplex Immunohistochemistry (IHC) and Immunofluorescence (IF) Staining and Validation. J immunother canc (2020) 8(1):e000155. doi: 10.1136/jitc-2019-000155
50. Francisco-Cruz A, Parra ER, Tetzlaff MT, Wistuba II. Multiplex Immunofluorescence Assays. Methods Mol Biol (2020) 2055:467–95. doi: 10.1007/978-1-4939-9773-2_22
51. Erlandsson A, Carlsson J, Lundholm M, Falt A, Andersson SO, Andren O, et al. M2 Macrophages and Regulatory T Cells in Lethal Prostate Cancer. Prostate (2019) 79(4):363–9. doi: 10.1002/pros.23742
52. Leo P, Janowczyk A, Elliott R, Janaki N, Bera K, Shiradkar R, et al. Computer Extracted Gland Features From H&E Predicts Prostate Cancer Recurrence Comparably to a Genomic Companion Diagnostic Test: A Large Multi-Site Study. NPJ Precis Oncol (2021) 5(1):35. doi: 10.1038/s41698-021-00174-3
53. Vitkin N, Nersesian S, Siemens DR, Koti M. The Tumor Immune Contexture of Prostate Cancer. Front Immunol (2019) 10. doi: 10.3389/fimmu.2019.00603
54. Nagarajan D, McArdle SEB. Immune Landscape of Breast Cancers. Biomedicines (2018) 6(1):20. doi: 10.3390/biomedicines6010020
55. Landskron G, de la Fuente M, Thuwajit P, Thuwajit C, Hermoso MA. Chronic Inflammation and Cytokines in the Tumor Microenvironment. J Immunol Res (2014) 2014:149185. doi: 10.1155/2014/149185
56. Yuan X, Wang J, Huang Y, Shangguan D, Zhang P. Single-Cell Profiling to Explore Immunological Heterogeneity of Tumor Microenvironment in Breast Cancer. Front Immunol (2021) 12. doi: 10.3389/fimmu.2021.643692
57. Campbell MJ, Scott J, Maecker HT, Park JW, Esserman LJ. Immune Dysfunction and Micrometastases in Women With Breast Cancer. Breast Cancer Res Treat (2005) 91(2):163–71. doi: 10.1007/s10549-004-7048-0
58. Nederlof I, Horlings HM, Curtis C, Kok M. A High-Dimensional Window Into the Micro-Environment of Triple Negative Breast Cancer. Cancers (2021) 13(2):316. doi: 10.3390/cancers13020316
59. Burstein HJ, Curigliano G, Loibl S, Dubsky P, Gnant M, Poortmans P, et al. Estimating the Benefits of Therapy for Early-Stage Breast Cancer: The St. Gallen International Consensus Guidelines for the Primary Therapy of Early Breast Cancer 2019. Ann Oncol (2019) 30(10):1541–57. doi: 10.1093/annonc/mdz235
60. Cardoso F, Kyriakides S, Ohno S, Penault-Llorca F, Poortmans P, Rubio IT, et al. Early Breast Cancer: ESMO Clinical Practice Guidelines for Diagnosis, Treatment and Follow-Up†. Ann Oncol (2019) 30(8):1194–220. doi: 10.1093/annonc/mdz173
61. Loi S, Michiels S, Adams S, Loibl S, Budczies J, Denkert C, et al. The Journey of Tumor-Infiltrating Lymphocytes as a Biomarker in Breast Cancer: Clinical Utility in an Era of Checkpoint Inhibition. Ann Oncol (2021) 32(10):1236–44. doi: 10.1016/j.annonc.2021.07.007
62. Park JH, Jonas SF, Bataillon G, Criscitiello C, Salgado R, Loi S, et al. Prognostic Value of Tumor-Infiltrating Lymphocytes in Patients With Early-Stage Triple-Negative Breast Cancers (TNBC) Who did Not Receive Adjuvant Chemotherapy. Ann Oncol (2019) 30(12):1941–9. doi: 10.1093/annonc/mdz395
63. De Jong VMT, Wang Y, Opdam M, Nt H, Jóźwiak K, Hauptmann M, et al. 159o Prognostic Value of Tumour Infiltrating Lymphocytes in Young Triple Negative Breast Cancer Patients Who did Not Receive Adjuvant Systemic Treatment; by the PARADIGM Study Group. Ann Oncol (2020) 31:S303. doi: 10.1016/j.annonc.2020.08.281
64. Boccignone G, Napoletano P, Caggiano V, Ferraro M. A Multiresolution Diffused Expectation-Maximization Algorithm for Medical Image Segmentation. Comput Biol Med (2007) 37(1):83–96. doi: 10.1016/j.compbiomed.2005.10.002
65. Oyama T, Allsbrook WC Jr., Kurokawa K, Matsuda H, Segawa A, Sano T, et al. A Comparison of Interobserver Reproducibility of Gleason Grading of Prostatic Carcinoma in Japan and the United States. Arch Pathol Lab Med (2005) 129(8):1004–10. doi: 10.5858/2005-129-1004-ACOIRO
66. Metrock LK, Summers RJ, Park S, Gillespie S, Castellino S, Lew G, et al. Utility of Peripheral Blood Immunophenotyping by Flow Cytometry in the Diagnosis of Pediatric Acute Leukemia. Pediatr Blood Cancer (2017) 64(10):e26526. doi: 10.1002/pbc.26526
67. Taussig MJ, Fonseca C, Trimmer JS. Antibody Validation: A View From the Mountains. New Biotechnol (2018) 45:1–8. doi: 10.1016/j.nbt.2018.08.002
68. Tsujikawa T, Kumar S, Borkar RN, Azimi V, Thibault G, Chang YH, et al. Quantitative Multiplex Immunohistochemistry Reveals Myeloid-Inflamed Tumor-Immune Complexity Associated With Poor Prognosis. Cell Rep (2017) 19(1):203–17. doi: 10.1016/j.celrep.2017.03.037
69. Fukumura K, Malgulwar PB, Fischer GM, Hu X, Mao X, Song X, et al. Multi-Omic Molecular Profiling Reveals Potentially Targetable Abnormalities Shared Across Multiple Histologies of Brain Metastasis. Acta Neuropathol (2021) 141(2):303–321. doi: 10.1007/s00401-020-02256-1
70. Parra ER, Ferrufino-Schmidt MC, Tamegnon A, Zhang J, Solis L, Jiang M, et al. Immuno-Profiling and Cellular Spatial Analysis Using Five Immune Oncology Multiplex Immunofluorescence Panels for Paraffin Tumor Tissue. Sci Rep (2021) 11(1):8511. doi: 10.1038/s41598-021-88156-0
71. Price JH, Goodacre A, Hahn K, Hodgson L, Hunter EA, Krajewski S, et al. Advances in Molecular Labeling, High Throughput Imaging and Machine Intelligence Portend Powerful Functional Cellular Biochemistry Tools. J Cell Biochem Suppl (2002) 39:194–210. doi: 10.1002/jcb.10448
72. Camp RL, Chung GG, Rimm DL. Automated Subcellular Localization and Quantification of Protein Expression in Tissue Microarrays. Nat Med (2002) 8(11):1323–7. doi: 10.1038/nm791
73. Ginty F, Adak S, Can A, Gerdes M, Larsen M, Cline H, et al. The Relative Distribution of Membranous and Cytoplasmic Met is a Prognostic Indicator in Stage I and II Colon Cancer. Clin Cancer Res (2008) 14(12):3814–22. doi: 10.1158/1078-0432.CCR-08-0180
74. Viratham Pulsawatdi A, Craig SG, Bingham V, McCombe K, Humphries MP, Senevirathne S, et al. A Robust Multiplex Immunofluorescence and Digital Pathology Workflow for the Characterisation of the Tumour Immune Microenvironment. Mol Oncol (2020) 14(10):2384–402. doi: 10.1002/1878-0261.12764
75. Hamid O, Robert C, Daud A, Hodi FS, Hwu WJ, Kefford R, et al. Safety and Tumor Responses With Lambrolizumab (Anti-PD-1) in Melanoma. N Engl J Med (2013) 369(2):134–44. doi: 10.1056/NEJMoa1305133
76. Provencio M, Nadal E, Insa A, Garcia-Campelo MR, Casal-Rubio J, Domine M, et al. Neoadjuvant Chemotherapy and Nivolumab in Resectable non-Small-Cell Lung Cancer (NADIM): An Open-Label, Multicentre, Single-Arm, Phase 2 Trial. Lancet Oncol (2020) 21(11):1413–22. doi: 10.1016/S1470-2045(20)30453-8
77. Morihiro T, Kuroda S, Kanaya N, Kakiuchi Y, Kubota T, Aoyama K, et al. PD-L1 Expression Combined With Microsatellite Instability/CD8+ Tumor Infiltrating Lymphocytes as a Useful Prognostic Biomarker in Gastric Cancer. Sci Rep (2019) 9(1):4633. doi: 10.1038/s41598-019-41177-2
78. Lee CW, Ren YJ, Marella M, Wang M, Hartke J, Couto SS. Multiplex Immunofluorescence Staining and Image Analysis Assay for Diffuse Large B Cell Lymphoma. J Immunol Methods (2020) 478:112714. doi: 10.1016/j.jim.2019.112714
79. Shipitsin M, Small C, Giladi E, Siddiqui S, Choudhury S, Hussain S, et al. Automated Quantitative Multiplex Immunofluorescence in Situ Imaging Identifies Phospho-S6 and Phospho-PRAS40 as Predictive Protein Biomarkers for Prostate Cancer Lethality. Proteome Sci (2014) 12:40. doi: 10.1186/1477-5956-12-40
80. Laberiano-Fernandez C, Hernandez-Ruiz S, Rojas F, Parra ER. Best Practices for Technical Reproducibility Assessment of Multiplex Immunofluorescence. Front Mol Biosci (2021) 8:660202. doi: 10.3389/fmolb.2021.660202
81. Velcheti V, Schalper KA, Carvajal DE, Anagnostou VK, Syrigos KN, Sznol M, et al. Programmed Death Ligand-1 Expression in non-Small Cell Lung Cancer. Lab Invest (2014) 94(1):107–16. doi: 10.1038/labinvest.2013.130
82. Parra ER, Villalobos P, Behrens C, Jiang M, Pataer A, Swisher SG, et al. Effect of Neoadjuvant Chemotherapy on the Immune Microenvironment in non-Small Cell Lung Carcinomas as Determined by Multiplex Immunofluorescence and Image Analysis Approaches. J Immunother Cancer (2018) 6(1):48. doi: 10.1186/s40425-018-0368-0
83. Hernandez S, Rojas F, Laberiano C, Lazcano R, Wistuba I, Parra ER. Multiplex Immunofluorescence Tyramide Signal Amplification for Immune Cell Profiling of Paraffin-Embedded Tumor Tissues. Front Mol Biosci (2021) 8:667067. doi: 10.3389/fmolb.2021.667067
84. Pratapa A, Doron M, Caicedo JC. Image-Based Cell Phenotyping With Deep Learning. Curr Opin Chem Biol (2021) 65:9–17. doi: 10.1016/j.cbpa.2021.04.001
85. Parra ER, Jiang M, Solis L, Mino B, Laberiano C, Hernandez S, et al. Procedural Requirements and Recommendations for Multiplex Immunofluorescence Tyramide Signal Amplification Assays to Support Translational Oncology Studies. Cancers (Basel) (2020) 12(2):255. doi: 10.3390/cancers12020255
86. Doyle S, Madabhushi A, Feldman M, Tomaszeweski J. A Boosting Cascade for Automated Detection of Prostate Cancer From Digitized Histology. Med Image Comput Comput Assist Interv (2006) 9(Pt 2):504–11. doi: 10.1007/11866763_62
87. Demir C, Gultekin SH, Yener B. Augmented Cell-Graphs for Automated Cancer Diagnosis. Bioinformatics (2005) 21 Suppl 2:ii7–12. doi: 10.1093/bioinformatics/bti1100
88. Jawhar NMT. Tissue Microarray: A Rapidly Evolving Diagnostic and Research Tool. Ann Saudi Med (2009) 29(2):123–7. doi: 10.4103/0256-4947.51806
89. Lee SL, Cabanero M, Hyrcza M, Butler M, Liu F-F, Hansen A, et al. Computer-Assisted Image Analysis of the Tumor Microenvironment on an Oral Tongue Squamous Cell Carcinoma Tissue Microarray. Clin Trans Radiat Oncol (2019) 17:32–9. doi: 10.1016/j.ctro.2019.05.001
90. Mercan E, Aksoy S, Shapiro LG, Weaver DL, Brunyé TT, Elmore JG. Localization of Diagnostically Relevant Regions of Interest in Whole Slide Images: A Comparative Study. J Digital Imaging (2016) 29(4):496–506. doi: 10.1007/s10278-016-9873-1
91. Romo D, Romero E, González F. Learning Regions of Interest From Low Level Maps in Virtual Microscopy. Diagn Pathol (2011) 6 Suppl 1(Suppl 1):S22. doi: 10.1186/1746-1596-6-S1-S22
92. Huang C-H, Veillard A, Roux L, Loménie N, Racoceanu D. Time-Efficient Sparse Analysis of Histopathological Whole Slide Images. Computerized Med Imaging Graphics (2011) 35(7):579–91. doi: 10.1016/j.compmedimag.2010.11.009
93. Moen E, Bannon D, Kudo T, Graf W, Covert M, Van Valen D. Deep Learning for Cellular Image Analysis. Nat Methods (2019) 16(12):1233–46. doi: 10.1038/s41592-019-0403-1
94. Bandi P, Balkenhol M, van Ginneken B, van der Laak J, Litjens G. Resolution-Agnostic Tissue Segmentation in Whole-Slide Histopathology Images With Convolutional Neural Networks. PeerJ (2019) 7:e8242. doi: 10.7717/peerj.8242
95. Sinaga KP, Yang M. Unsupervised K-Means Clustering Algorithm. IEEE Access (2020) 8:80716–27. doi: 10.1109/ACCESS.2020.2988796
96. Zhu Q, Tang X, Elahi A. Application of the Novel Harmony Search Optimization Algorithm for DBSCAN Clustering. Expert Syst Applications (2021) 178:115054. doi: 10.1016/j.eswa.2021.115054
97. Parra ER, Uraoka N, Jiang M, Cook P, Gibbons D, Forget MA, et al. Validation of Multiplex Immunofluorescence Panels Using Multispectral Microscopy for Immune-Profiling of Formalin-Fixed and Paraffin-Embedded Human Tumor Tissues. Sci Rep (2017) 7(1):13380. doi: 10.1038/s41598-017-13942-8
98. Friedman J, Hastie T, Tibshirani R. Regularization Paths for Generalized Linear Models via Coordinate Descent. J Stat Software (2010) 33(1):1–22. doi: 10.18637/jss.v033.i01
99. Chennubhotla C, Clarke LP, Fedorov A, Foran D, Harris G, Helton E, et al. An Assessment of Imaging Informatics for Precision Medicine in Cancer. Yearb Med Inform (2017) 26(1):110–9. doi: 10.15265/IY-2017-041
100. Muñoz-Aguirre M, Ntasis VF, Guigó R. PyHIST: A Histological Image Segmentation Tool. bioRxiv (2020) 2020:082461. doi: 10.1371/journal.pcbi.1008349
101. Hofman P, Badoual C, Henderson F, Berland L, Hamila M, Long-Mira E, et al. Multiplexed Immunohistochemistry for Molecular and Immune Profiling in Lung Cancer-Just About Ready for Prime-Time? Cancers (Basel) (2019) 11(3):283. doi: 10.3390/cancers11030283
102. Yue L, Tian D, Chen W, Han X, Yin M. Deep Learning for Heterogeneous Medical Data Analysis. World Wide Web (2020) 23(5):2715–37. doi: 10.1007/s11280-019-00764-z
103. Yuste R, Goering S, Arcas BAY, Bi G, Carmena JM, Carter A, et al. Four Ethical Priorities for Neurotechnologies and AI. Nature (2017) 551(7679):159–63. doi: 10.1038/551159a
104. Bibbo M, Kim DH, Pfeifer T, Dytch HE, Galera-Davidson H, Bartels PH. Histometric Features for the Grading of Prostatic Carcinoma. Anal Quant Cytol Histol (1991) 13(1):61–8.
105. Markiewicz T, Osowski S, Patera J, Kozlowski W. Image Processing for Accurate Cell Recognition and Count on Histologic Slides. Anal Quant Cytol Histol (2006) 28(5):281–91.
106. Belien JA, Baak JP, van Diest PJ, van Ginkel AH. Counting Mitoses by Image Processing in Feulgen Stained Breast Cancer Sections: The Influence of Resolution. Cytometry (1997) 28(2):135–40. doi: 10.1002/(SICI)1097-0320(19970601)28:2<135::AID-CYTO6>3.0.CO;2-E
Keywords: immunotherapy, multiplex immunofluorescence, image analysis algorithms, tissue, machine learning
Citation: Rojas F, Hernandez S, Lazcano R, Laberiano-Fernandez C and Parra ER (2022) Multiplex Immunofluorescence and the Digital Image Analysis Workflow for Evaluation of the Tumor Immune Environment in Translational Research. Front. Oncol. 12:889886. doi: 10.3389/fonc.2022.889886
Received: 04 March 2022; Accepted: 27 May 2022;
Published: 27 June 2022.
Edited by:
William L. Redmond, Earle A. Chiles Research Institute, United StatesReviewed by:
Rasha Abu Eid, University of Aberdeen, United KingdomEwan Millar, St. George Hospital, Australia
Renee Whan, University of New South Wales, Australia
Copyright © 2022 Rojas, Hernandez, Lazcano, Laberiano-Fernandez and Parra. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Edwin Roger Parra, ZXJwYXJyYUBtZGFuZGVyc29uLm9yZw==