I think the more convincing argument is that most known applications of quantum computers (sidestepping any hardware practicalities), are for niche problems (in my wheelhouse, quantum simulation), the average person has no (practically advantageous) reason to own a quantum computer.
I suspect that once quantum computers actually scale up so that you can play with them, we'll find all sorts of interesting things to do with them.
However, even now, you can imagine that if quantum computers were small enough, it would be worth it to have it just for the asymptotically fast prime generation with Shor's algorithm. I don't think that's that far fetched. Of course, people wouldn't necessarily need to know they have a quantum computer, but they don't necessarily know the workings of their computers today anyway.
