A better analogy would be that people who would be afraid to try goat's milk are probably the same type of people who would be afraid to eat venison (deer meat). My brother used to hunt and bring home deer, and we'd go to the butcher to have it prepared, and we'd eat it for months afterwards. Venison is the best meat that I've ever eaten, and I wish that my brother still liked to hunt so that I could eat it every year. Our older sister, however, never tried it because she didn't trust anything that didn't come from the store, which is pretty backwards if you think about the kind of conditions that cows live in before being slaughtered vs. eating meat from a wild, free-range animal.
Again, no one thinks twice about eating from a cow, but anything more uncommon can cause people to hesitate.