Yes, I have spent a lot of time out in nature and parks, and I generally am aware of not leaving a trace or litter. However, with something like a banana peel, I'd rather it end up as tree food the in a landfill, so I usually toss them in bushes.

The issue is that leaving food for the animals alters their behavior and can cause issues. You're probably not the worst offender wherever you are, as there are many people who just give food to animals, but the argument would be that it contributes to the problem.

Actually banana peels and orange peels don't biodegrade that fast in a north American forest.

Apple cores I think might not be too bad to toss, ants can eat that up in a day or two.

I pack out the peels of any tropical fruit.

Please don't do that with banana or orange peels, they remain present for a long time. I have no problem personally with people tossing apple cores in more remote, little-visited areas (though it still makes sense to crack down on such things a bit more in heavily visited areas like National Parks) but various peels don't rot well and remain in place.