Here's a fantastic talk that describes a way to structure one's approach to this explore/exploit tradeoff that draws on research in areas like search algorithms, cognitive science, etc.

https://www.youtube.com/watch?v=j0XmixCsWjs