I used to (I was an unidentified Aspie back then, now diagnosed), and my eldest daughter (NT, not Aspie) currently does.
I found Sydney University very difficult, because I felt lost in the sea of faces and the lack of clear structures. I coped with high school, but fell apart at uni.
Nevertheless, I was a student and postgrad there for many years. Fisher Library is great for pursuing your special interests, and I have been back to Fisher since I finished being a student at Sydney.