I would never assume anything without a direct invitation, but its probably a good idea to be prepared, just in case. If its just a farewell celebration for the person who's leaving, they may be expecting you to stay behind and answer phones while they're gone.
I always tried to politely decline when coworkers did things like that. Sitting around socializing with a group of people most of whom I don't really know (or care to), when I have more constructive things I could be accomplishing just makes me anxious and irritable. Of course not participating in their cliquish ritual makes them think you're being intentionally rude, or at least cold and aloof. Ironically, if you go, they'll ignore you the whole time and chat merrily with each other, just like they do back at work. 