Cache is also an extremely limited resource and the more you waste the less other programs have to work with, slowing them down too, some of what your program pushes out of the cache could be slowing down disk and network processing. There's a tragedy of the commons effect happening in our CPU's.

That doesn't mean it's always worth otimising for, but it's worth remembering yours isn't the only process running.