Alzheimer’s disease (AD) is the most common neurodegenerative disease characterized by progressive memory loss. Although AD neuropathological hallmarks are extracellular amyloid plaques and intracellular tau tangles, the best correlate of disease progression is synapse loss. What causes synapse loss has been the focus of several researchers in the AD field. Synapses become dysfunctional before plaques and tangles form. Studies based on early-onset familial AD (eFAD) models have supported that synaptic transmission is depressed by β-amyloid (Aβ) triggered mechanisms. Since eFAD is rare, affecting only 1% of patients, research has shifted to the study of the most common late-onset AD (LOAD). Intracellular trafficking has emerged as one of the pathways of LOAD genes. Few studies have assessed the impact of trafficking LOAD genes on synapse dysfunction. Since endocytic traffic is essential for synaptic function, we reviewed Aβ-dependent and independent mechanisms of the earliest synaptic dysfunction in AD. We have focused on the role of intraneuronal and secreted Aβ oligomers, highlighting the dysfunction of endocytic trafficking as an Aβ-dependent mechanism of synapse dysfunction in AD. Here, we reviewed the LOAD trafficking genes APOE4, ABCA7, BIN1, CD2AP, PICALM, EPH1A, and SORL1, for which there is a synaptic link. We conclude that in eFAD and LOAD, the earliest synaptic dysfunctions are characterized by disruptions of the presynaptic vesicle exo- and endocytosis and of postsynaptic glutamate receptor endocytosis. While in eFAD synapse dysfunction seems to be triggered by Aβ, in LOAD, there might be a direct synaptic disruption by LOAD trafficking genes. To identify promising therapeutic targets and biomarkers of the earliest synaptic dysfunction in AD, it will be necessary to join efforts in further dissecting the mechanisms used by Aβ and by LOAD genes to disrupt synapses.