That depends entirely on your window manager. I generally run a customized Fvwm2 setup, so that's all easy to tweak.

In my setup, focus follows mouse only when the mouse moves

That makes much more sense to me.