As it's cached, it's only the first load that will have the lag. I'd guess it will be quite small as well. If the CSS makes heavy use of mixins and the nesting ability then the lag may even be smaller than the time to transfer the compiled CSS.
First load is where it counts! This is when users first see your site: their first impression. This is an important time that should be kept as fast as possible. Subsequent loads, for most sites, are likely not as important speed wise.
Remember, we have to design our sites as if the user is poised over the back button.
Maybe.