Pour the potatoes into a large bowl. Mash by hand with a potato masher. Then stir in the brown sugar, butter, salt, nutmeg, and vanilla. Finally add the milk and blend with a hand mixer until smooth. Transfer to the baking dish.
1/2 cup brown sugar, 1/4 cup unsalted butter, 1/2 teaspoon salt, 1/4 teaspoon nutmeg, 1/2 teaspoon vanilla extract, 1/2 cup whole milk