I'm having a bit of a hard time following who is whom in your story, so my response is based on this assumption:
You asked your hostess for the recipe of her sister's potato dish instead of her (the hostess's) own.
If that's the case, it sounds like the hostess is being snitty that people didn't like her dish and preferred her sister's. She's probably embarrassed that hers didn't come out as well or was just an inferior recipe.
I would probably not have asked for the recipe at all (though you did the correct thing, etiquette-wise, by asking the hostess since it's her table) because I would have anticipated the discomfort that would have arisen from two such-similar dishes having been served. If the sister had made something completely different than the hostess, the hostess would probably have been fine with procuring the recipe for you.
Or, maybe your hostess is just a noob at entertaining and doesn't realize that it's her responsibility to get the recipe for you. She could be sitting there puzzled, wondering why you didn't just directly approach the maker of the dish.
It used to be one never complimented the food, one complimented the hostess on having (and managing) such a fine chef. How times change!