what strikes me about your post is that i don't think you should let yourself feel inferior or compare yourself to others in terms of number of friends and amount of time spent socialising? You sound very much like a school-aged me, (although I don't know your age, and i apologise if I have made the wrong assumption) I used to really feel like something was wrong with me then, and that my lack of socialising was something to be ashamed of.
In terms of having a social circle, what do YOU want? (ignoring whatever everyone else thinks you ought to want). You would probably like a few more friends, so that you have someone to see when your current friends are busy, but would you really want a very big circle of friends? A lot of people with aspergers, (including myself!) find that far too exhausting! Nothing wrong with that. But the point is that it's about having a group of friends that meets your needs rather than worrying about what anyone else thinks of you.
The good thing about being downright crazy is that at the very least you will get remembered. That's a lot better than being bland. I promise you it can be an advantage sometimes. And not everyone wants friends who are loud and have to attend every single party either.
Your situation does sound horrible, I hope you manage to find more friends, but just don't let other people judge you!! ! It's not their business!!