Over the past decades, a growing body of evidence has demonstrated the impact of prenatal environmental adversity on the development of the human embryonic and fetal brain. Prenatal environmental adversity includes infectious agents, medication, and substances of use as well as inherently maternal factors, such as diabetes and stress. These adversities may cause long-lasting effects if occurring in sensitive time windows and, therefore, have high clinical relevance. However, our knowledge of their influence on specific cellular and molecular processes of in utero brain development remains scarce. This gap of knowledge can be partially explained by the restricted experimental access to the human embryonic and fetal brain and limited recapitulation of human-specific neurodevelopmental events in model organisms. In the past years, novel 3D human stem cell-based in vitro modeling systems, so-called brain organoids, have proven their applicability for modeling early events of human brain development in health and disease. Since their emergence, brain organoids have been successfully employed to study molecular mechanisms of Zika and Herpes simplex virus-associated microcephaly, as well as more subtle events happening upon maternal alcohol and nicotine consumption. These studies converge on pathological mechanisms targeting neural stem cells. In this review, we discuss how brain organoids have recently revealed commonalities and differences in the effects of environmental adversities on human neurogenesis. We highlight both the breakthroughs in understanding the molecular consequences of environmental exposures achieved using organoids as well as the on-going challenges in the field related to variability in protocols and a lack of benchmarking, which make cross-study comparisons difficult.