The first rule when you going to say something--anything--is, "know your audience."
The type of behavior that is perfectly acceptable among your friends and your colleagues might be very offensive to people who you don't know, and who don't know you. While you might believe that your words are meant in fun, another person might take them very personally.
If you aren't sure about the people who can hear you, the best option is to exercise prudence.
And here is where the people involved failed. They might have intended no harm or derision with their use of the word, "fag." But their intention means nothing when harm or derision results.
_________________
--James