Good point, but to each his own. I generally don't keep my car anywhere near dirty for more than 24 hours... even in snowy times...
It's really not a matter of dirt.
Over time, the paint will oxidize naturally by being exposed to air, sun and any chemicals that may be on the road, however brief the exposure is. Because there's something on top of the paint (the sticker), it means that the exposure process won't happen evenly, and when the sticker is taken off, there may be a difference in the finish.
I mean, this doesn't happen overnight. It happens over a couple of years and can certainly be minimized through waxing and otherwise taking care of the finish. None the less, it's not 100% preventable over
long periods of time. It'll be more significant in harsher climates, such as the temp and the blazing sun of the south, where clearcoat takes a heavy beating.