Combine sugar, cornstarch, and milk in a medium saucepan over medium heat. Stir frequently until pudding heats up, then stir constantly until it starts to bubble.
In a small bowl add the egg yolks and whisk. Pour 1/2 cup of the milk mixture into the eggs stirring constantly. You don’t want to cook your egg yolks.
Add the pumpkin, cinnamon, vanilla, and nutmeg into the bowl with the yolk and milk mixture.
Then pour the egg mixture back into the saucepan, stir well to combine and cook until bubbly and thickened.
Pour into a bowl and cover with plastic wrap to prevent skin from forming. Place in the refrigerator for at least an hour.