Previous research has identified ventral and dorsal white matter tracts as being crucial for language processing; their maturation correlates with increased language processing capacity. Unknown is whether the growth or maintenance of these language-relevant pathways is shaped by language experience in early life. To investigate the effects of early language deprivation and the sensory-motor modality of language on white matter tracts, we examined the white matter connectivity of language-relevant pathways in congenitally deaf people with or without early access to language. We acquired diffusion tensor imaging (DTI) data from two groups of individuals who experienced language from birth, twelve deaf native signers of American Sign Language, and twelve hearing L2 signers of ASL (native English speakers), and from three, well-studied individual cases who experienced minimal language during childhood. The results indicate that the sensory-motor modality of early language experience does not affect the white matter microstructure between crucial language regions. Both groups with early language experience, deaf and hearing, show leftward laterality in the two language-related tracts. However, all three cases with early language deprivation showed altered white matter microstructure, especially in the left dorsal arcuate fasciculus (AF) pathway.