I think they will be okay. When they grow up it might be weird telling their friends that they can't have music in the house or something, but honestly, kids "suffer" much worse things than this. A lot of people don't like crowds, not just ASD.
If their are other people in their lives, NT parent, grandparents, aunts, etc then they can hang out with them some as they get older. If your kids go to school they will experience music and crowds there.
Just do the best you can. It's all any of us can do. And I know that's not a good enough answer, and it won't stop you from worrying, but really, what more can you do? Worry about your kids and love them.