You are grossly oversimplifying a complicated process of digesting food and its interaction with the overal metabolism of the body. Yes, calories play a factor, but also in what form you ingest them. Then there's genetic factors, gut bacteria, physical exercise and probably a whole slew of other things that interact with each other.