Can the land actually handle building up? Wouldn't building down be the better way to go? I mean most of those cities are already threatened by rising sea levels. Boston and New York are pretty dense areas who's home do you want to tear down to build new more expensive high rises that the working poor can't afford.
Skyscrapers go straight down into bedrock - the Prudential and Hancock in Boston are built on landfill in the Back Bay, but it doesn't matter, because their foundations go way past the mud. There are a few architectural/area combinations you want to avoid (you probably don't want 4-story rowhouses in the Marina in SF, for example), but most of the land area in these cities is solid. Western SF, for example, is actually at a high elevation immune to most reasonable sea level rise projections:
(The East Bay and Palo Alo are screwed, but that's not the subject here.)
Also, expensive high rises benefit the working poor by giving rich yuppies a place to live where they don't bid up the same housing that working-class people have been living in for decades. If you put up luxury condo highrises in SOMA where tech workers could walk to their offices, for example, then maybe they wouldn't be bidding up the price of 1BRs in the Mission to $4k/month. Landlords are generally unwilling to let housing go vacant, because even a 2-month vacancy can destroy much of their profit for the year. So one way or another, if you build enough housing the price will adjust itself to the point where everybody gets housing.
Back Bay of Boston is really a marvel. I have literally no idea about California so I'll have to take your word for it. My concern is that the only people who seen willing to build are those who stand to make the most money. It would seem that the effective way to do this would be to require as part of codes for building different cost zoning to be taken into account. When I was in Tokyo they had elevators sectioned off by floor access, which would provide the ability to have luxury, shopping, and lower/mid income mixed into the same building. When I've traveled to Europe I've been interested in how businesses have been lower levels with hosting above, something I've not seen much of in the US.
These cities are only dense in the American sense. Even Manhattan looks unremarkable next to a properly dense city like Sao Paulo. Los Angeles is probably the best city suited for becoming really dense. The basin is sprawling, it can fit over 20 manhattan islands, but most of it is low slung single family homes that could one day be upzoned into towers. Lots of areas near downtown are just industrial/warehouses too and could be converted into housing in the future.
It has to happen eventually. As long as there is economic activity, there will be jobs hiring the best talent from around the world, spurs more economic activity, etc. That's just not going to change, demand will continue, so cities really need to get rid of zoning for height so supply could be built near the job centers. The alternative is sprawl, which means people need to be transported, consuming resources, etc. Infrastructure is also all the more difficult to build out over sprawl. In particular for California, adding to the sprawl could also mean building into wildfire prone areas.
Correct my if I'm wrong, but don't a lot of the ultra-dense cities have some form of areas of makeshift or very unhealthy living situations? Sao Paulo was a city like this, as are many in ultra dense places in Asia, Tokyo excluded but they built down as well as up and routinely demolish relatively new buildings.
California really does have a major problem now with wildfires and encroachment. I was amazed how fast the Camp Fire moved (an acre a second burnt at one point). There's just no way to really escape something like that. This is why I think building down as well as up is a good idea for many cities, Californian cities may have dinner different issues due to their tectonic activity but it should at least be explored.
I agree with your point regarding density in general. I always find it silly when people say America doesn't have space for more people. We have entire states with less population than some cities. I think improving infrastructure between the "flyover" states and coastal states might be one improvement that's overdue, but I don't know what the best solution would be off the top of my head.
