There used to be a tradition of cutting down seed heads and spent perennial plants in late Autumn. Whilst this did ensure the garden looked ‘tidy’, it also meant that it could look quite bare through Winter until maybe the Spring bulbs started. There has been a movement in recent years to leave more plant material around as it can look quite beautiful covered in frost or even snow!
Additionally things like seed heads often provide a valuable source of food for birds as well as material and even hiding or nesting places for other mammals. Alliums, eryngium, echinops, nigella and poppies all have great structural seed heads that look great in an autumn or winter garden, or brought in when dried as well as being a source of food whilst in the garden.
I certainly leave many of my allium heads right through winter, although admittedly some make it indoors for Christmas when they are sprayed silver or gold and make fantastic decorations! I have even used some in Winter and Spring flower arrangements.