Lilies stink and do smell like decomposition as Magna describes. Furthermore, they're used in most funeral arrangements so I also associate their scent with death. I won't let the scent of lilies in my house. Any time I've received a wildflower bouquet I pull the lilies out first. It's possible to just remove the stamen but I'd rather not see lilies at all.
Note: They're even on the credits of SFU.