I'm not saying it isn't a problem, I just wanted to know if I was correct in thinking that it is only a problem the first time you compile each crate (or rather each version of each crate, I think).
Yes, it is a problem when you do a clean build, or when you have common crates across projects, because cargo doesn't have a concept of build cache.
You can try to workaround it by setting all target directories to same one via target-dir in your .cargo/config file, but there is no guarantee that the crate won't get rebuild.