I think it has more to do with behavior/dress than a specific bone structure or something physical. When I find myself looking at another person wondering if they might have Asperger's also (such as another person who works in my building) it is those kinds of traits. These are also the things related to appearance that people have commented about me as being weird:
"Dorky" clothing/no fashion sense - It's not that I don't care about what I wear. I do. I pick things that are comfortable and also that I like the appearance of. This just has never fit in with what is considered "fashionable". Also I don't believe in spending boatloads of money for brand names (or recognize most of them) or things like jeans that are already ripped (that's just illogical, sorry).
Peculiar gait: I have been told that I lope, I move my shoulders too much, walk too fast, clomp my feet, etc. Lots of complaints on this. I'm also fairly clumsy and run into things, and I tend to slouch (don't know if that's related).
Strange voice. I am told my voice is monotonous (when I used to work in a call center, people frequently thought I was an answering machine), or that I "swallow" my words making them hard to understand. I frequently get asked if I have an accent (I don't, I live in the same region of the country where I grew up). I do have a speech impediment (slight lisp) but I don't think this accounts for all of it. I used to get teased about the lisp a lot as a kid, when it was more pronounced (heheh), but people do not comment on it as much as an adult (to my face anyway). I have had people recognize me who have never seen me and only spoken on the phone, on the basis of my voice. Since I have no way of hearing my voice from the outside (and I'm told that everyone thinks their own voice sounds odd when recorded and played back) I can only base this on the NUMEROUS comments other people have made.
I also get a lot of comments about my speech patterns other than voice, that I use unusual or archaic words or strange constructions. People can't/don't usually explain what they mean by this, except that its "typical" of me. This part doesn't bother me, because I do like unusual words.
Lack of expression: People I meet seem to be divided into two categories... those who think I have a blank stare/sad expression and can never tell what I am thinking, and seem to expect more reaction, and those who can read everything I am thinking in my face. I'm don't think I'm very good at "covering" my expressions as many people do to mask what they feel, but some people apparently don't see this and others do. I'm also not very good at inserting face expressions appropriately unless I'm concentrating on "reacting" to people when they talk (which I do when it's especially important, but not all the time because it's exhausting).