There were very few Neanderthals in Europe who had viable offspring with modern humans compared to the size of the (otherwise quite heterogenous) migration wave coming from Africa - and their DNA was obviously very similar to ours from the beginning. So it makes sense that the net impact on genetic diversity wasn’t very large, even if it is detectable.