In a 10-inch cast-iron or other oven-proof skillet, melt the butter with 3/4 cup of brown sugar over medium heat. Stir until the sugar is melted and smooth. Stir in the berries and peaches or nectarines. Remove from heat and set aside.
In a medium mixing bowl, stir together the polenta, flour, baking powder and salt.
In another bowl with an electric mixer, beat the eggs and remaining 3/4 cup sugar until light, about 4 minutes. Beat in olive oil and vanilla. Mix the dry ingredients into the batter, until just combined.
Spoon batter over the fruit and smooth out with a spatula. Bake until the cake springs back when touched, about 20 minutes.
Let cool in the pan 10 minutes. Run thin knife around edge of cake. Cover with a platter and invert. Serve warm with whipped cream.