For better or worse, people will often judge you and set expectations via a halo effect. If you're good with writing backend code they likely assume you are good at writing front end code and thus also good at UX.

I sometimes find that I need to reset expectations myself when a business person say for example asks me to create some vector art because I used to do web dev.

Be clear to your co-workers about where your strengths lie and where they would be more productively served by someone else.