“just” is a subtle pejorative and unless you’re trying to be insulting, it’s better to rephrase what you’re saying especially if you’re getting negative reactions you don’t want.