There's no one test that can tell you whether you have Aspergers or not. A diagnosis generally comes from a variety of things, including some tick-the-box screening tests like the AQ test, EQ test and RAADS-R, interviews of the person undergoing assessment, information from family/partners/friends (info from your parents is often important to establish you had symptoms before aged 5, but not always essential, and you might be unaware of things that people close to you have noticed). Then there are a number of assessment tools such as the ADOS which involves a series of individual tests which look at observable aspects of social communication and imagination etc (i.e. they get you to do things like read a story from a book with just pictures, solve a puzzle, tell a story using certain objects etc). None of these things are designed to be used as a stand alone test. For assessing an adult, it's also important an assessor takes into account that a) people may not show symptoms until social demands outstrip their abilities (e.g. it's quite common in girls to seem fairly okay in primary school and then begin to fall apart socially as they go into puberty and friendship becomes more complex and based on different things), and b) people learn as they grow up to 'mask' their symptoms and develop coping skills so an adult might be quite good at hiding their deficits.
'Specialists' can be wrong but no one can say how likely that is. If you don't trust it, get a second opinion?