And even with high-speed rail, a lot of distances in the US would still be brutally slow, even with direct point-to-point rail.
At some point, trying to go from NYC to DC, let alone St. Louis or Atlanta, takes a pretty long time via non-plane methods. In some ways, the big innovation of plane travel is that for major cities, non-stop flights are an option, so it's hard to beat the speed of it. There's never going to be a viable non-stop train from, say, Miami to DC, but if it stops in Jacksonville, Atlanta, Greenville, Winston-Salem, and Richmond, then you've added hours to the trip time in stops/slowing down, which loses people as well who don't want to spend the whole day on a train.
There's a breakpoint around 4-5 hours where travel doesn't eat your full day, and there's only so many routes where you could get under that duration and still have enough volume of people to take it to make it worth running the route.
Yes, that's true. I forgot that because it doesn't top my personal list of important distinctions, but certainly plenty of others would feel differently.