Preheat oven to 350 degrees. Put 4 tablespoons butter in a 9-inch round baking pan. Place the pan in the oven just long enough to melt the butter. Remove pan from oven; scatter 1/2 cup brown sugar evenly over the melted butter.
With a sharp knife, peel the grapefruit, removing all pith, and slice. Arrange the slices to cover the bottom of the pan.
In a bowl, stir together flour, baking powder, cinnamon, ginger, baking soda, and allspice. In a separate bowl, using an electric mixer, beat remaining 6 tablespoons butter until creamy. Add granulated sugar and remaining 1/4 cup brown sugar; beat until well combined. Beat in eggs one at a time. Beat in vanilla. Alternately add dry ingredients and milk to butter mixture, beginning and ending with dry ingredients.
Pour batter over grapefruit in pan. Bake 45 minutes or until a toothpick inserted in center comes out clean. Cool 5 minutes; turn cake out onto serving plate. Serve warm or at room temperature, with whipped cream if desired.