well, it depends. Are you performing any functions (catering, entertainment, whatever) that they'll miss? If there's 35 people, unless you're in management, they shouldn't mind too much.
I used to go to parties, and glom onto the one person I knew...terrible at mingling. Eventually I joined a band, and was up on stage, where it's a lot easier to deal with the crowd..
Parties are supposed to be fun. If you know that you won't have a good time, it's your call. Talk with some trusted co-workers, maybe in a general way, about the party. One of the party cliches is 'I loathe parties, dahling'.. you could do that. Or, maybe something comes up....just can't make it, how was the party?
don't know if this helps, but good luck