I think that for a long time, my answer would have been that I would go with him. For a really long time I was so desperate for friends/feeling accepted that I would have gone for it just for that reason. However, when I was 20 I ended up being "too nice" to some guy who just got paroled (I met him in church, so it wasn't like I was seeking his company - he just happenned to be where I was). Anyway, he'd follow me around and tell me how he hadn't had sex in all the years he was in prison and I started to freak out. He also started to follow me around places like my work and stuff.
Anyway, that whole thing really has nothing to do with a person I just met asking me to come over and drink. If it was someone I knew, I would. I get wasted with people I know all the time. But if you do get wasted, you put yourself in a vulnerable position. I would not want to be in a vulnerable position with someone I had just barely met because they could be a sex-crazed parolee (that's how the parolee story ties in). If you have your faculties about you, at least you have a chance to escape a dangerous situation if the person you don't know turns on you.