I have Asperger's, and Avoident Personality Disorder, and Executive Function Disorder. I am also very verbal--no childhood speech delay. People with Asperger's are often very verbal, and often don't have a childhood speech delay, so I think the doc wasn't very knowledgeable about spectrum disorders. For a more accurate diagnosis you really need to see someone who has experience with spectrum disorders. And it will help if you bring copies of your online test results with you. Check the General Autism forum here for that. At least one post article on that forum has links to online tests.
But whether he is right or wrong, you are always welcome here at WP! 
_________________
If a man does not keep pace with his companions, perhaps it is because he hears a different drummer.
Let him step to the music which he hears, however measured, or far away.--Henry David Thoreau