Typically the fruit of the plant contains a lot of chemical energy in the form of sugars. Plants that have been selectively bred for hardiness (thicker skin, longer ripening periods, etc.) expend more on those traits as opposed to making more chemical energy in the fruit.
And that is probably much closer to the real reason. (And I'm saying "probably" just to hedge since I don't truly know, but it seems very likely to me.)
But I sort of get a kick out of imagining a meta-physical level of "flavor per fruit" out there in the world.
Incidentally, part of what I wonder about gene engineering is whether we could keep most of the hardiness, etc, and focus very tightly on putting in more flavor compounds, instead of using the blunt stick of breeding and getting who knows what other characteristics correlated in with that. Of course it won't be free to the plant but we might still have some room for optimizing.
Typically the fruit of the plant contains a lot of chemical energy in the form of sugars. Plants that have been selectively bred for hardiness (thicker skin, longer ripening periods, etc.) expend more on those traits as opposed to making more chemical energy in the fruit.