If the whole team is remote that wouldn't be a problem. If all of them are in the office, and you are the odd man out, then you should worry about that.

I believe that working successfully remotely doesn't work for every company or team. They must be actively encouraging it, and they should be aware of engagement issues that arise from working remotely.