Metrics Matters: A Deep Assessment of Deep Learning CNN Method for ENSO Forecast

Mohammad Naisipour ¹ Iraj Saeedpanah ^1* Arash Adib ² Saghar Ganji ³

• ¹ University of Zanjan, Zanjan, Zanjan, Iran
• ² Shahid Chamran University of Ahvaz, Ahvaz, Khuzestan, Iran
• ³ Shiraz University, Shiraz, Fars, Iran

Recent advancements in machine learning (ML) have significantly propelled the field, enhancing performance across diverse applications, including climate science and physical oceanography. However, the critical importance of reproducibility and the recreation of these methods cannot be overstated, as they ensure the reliability and trustworthiness of results. In this study, we thoroughly reproduced the state-of-the-art CNN method originally published by Ham et al. in Nature for predicting the El Niño-Southern Oscillation (ENSO), meticulously documenting the entire process from the creation of oceans' anomaly maps to the post-processing of outputs. While the transformation of raw data from repositories into input anomaly maps is complex and inadequately described in the original article, we developed these maps from scratch to guarantee their validity and to provide a clear methodology for other researchers to replicate. Each component of the CNN model was constructed from the ground up on the Google Colaboratory platform, aiming for holistic reproducibility and facilitating accessibility for fellow researchers. Our comprehensive analysis of the CNN method, evaluated through six distinct metrics, revealed that each metric illuminates different facets of model performance, highlighting a notable disparity in CNN accuracy between the 20th and 21st centuries, which may indicate an overfitting issue; thus, we recommend employing generalization techniques to reconcile prediction accuracy between these two periods. Although the correlation metric suggested that the CNN method and the advanced 3D-Geoformer have similar skill levels, error-based metrics revealed that the 3D-Geoformer outperformed the CNN, which has an RMSE five times higher than that of the 3D-Geoformer for short-term predictions. This highlights the importance of using diverse metrics for a thorough evaluation of machine learning models. We identified latent climatologies within the input data that may falsely improve future predictions, a phenomenon also observed in other recent methodologies. Moreover, we noted a common misapplication of historical CMIP runs for training machine learning models, as scientists frequently use data that overlaps with validation datasets, potentially leading to a misleading increase in model skill by as much as 6%. Finally, we achieved a 10 percent reduction in model execution time on Google Colaboratory without compromising accuracy.