Your comment seems to assume that your use case “big data engineer” is the only use case or the only Java use case and that the only context in which Rust can grow is to rewrite the existing tools that you use today (as opposed to greenfield projects like firecracker or bottle rocket). Your edit also writes Rust off as “idealistic” and not “pragmatic” without support, which is very far from the impression I have (based on my usage and seeing it used in real applications). In general, your comment seems like generic programming language flamebait.
Firecracker and Bottlerocket are fine pieces of engineering. If Rust wants to get a share of the pie when it comes to data engineering, then it needs to have at least a killer app with an ecosystem around it. We can talk all day and all night for why some languages are more successful than others but it seems like ecosystems are giving them a boost. Hopefully, Rust will be the language for microVMs and kernel-level modules. I was strictly referring to people that think Rust would displace Java as idealists.
I generally agree--languages do better when they become the language of a particular ecosystem and that ecosystem becomes relevant. Go had this with containers and network software a bit more broadly.
That said, I don't think anyone was suggesting Rust would displace Java. The OP was pretty clear IMO that his firm had experienced exponential growth in interest in Rust skills while he had been hoping that interest in Java would grow instead. This doesn't mean that Rust will replace Java for data engineering or any other domain in particular.
I can't absolutely say that this won't happen, but given Rust's focus on writing "correct" code, e.g. the borrow checker, etc, it's really unlikely given that one of the largest areas where Python and R shine is in exploratory data analysis. I can't imagine trying to play around with unstructured exploration of a data set in Rust's (totally reasonable) syntactic verbosity, correctness straitjackets, and compilation time.
Some of that complexity can be abstracted away with a good framework. I think it’s almost inevitable that a Pandas-like framework will eventually emerge for Rust (probably based on Arrow).
It’s also quite possible that Rust could underpin a product like Spark, with a scripting language on top.
The compilation time is a hurdle, but it’s not as if compiled languages are absent in numerical computing, for example Fortran.
Moreover, Logo-based and Lisp programming environments are more popular than other people think. Netlogo is heavily used in social sciences. Hard to think that even Python and R can displace those since they are so well suited for that kind of programming (agent manipulation).
