I must be doing something wrong because incremental builds regularly take 30-60 seconds for me. Much more if I add a dependency. And I try to keep my crates small.
As a sibling comment points out, it's likely to be mostly link time, not compilation time.
The most recent Rust version ships with `lld` so it shouldn't be the case anymore (afaik `lld` is a bit slower than the `mold` linker, but it's close, much closer than the system linker that was previously being used by default).