The important part is having physical access to the phone. A targeted attack against you now requires a physical element, rather than being entirely online.
Agree with everything you say, but add to that a lot of sms 2fa exploits are sim or redirection attacks. It’s possible to get access to a phone number without access to the phone.
Here’s an old story of a friend who had a weird talk with someone who had redirected their phone:
