From what I understand, for most NTs, it's the form that's important--the back and forth, the enjoyment of each other's company--rather than the content. My husband, who is very intelligent and likes to have a focused discussion as much as I do, also says that when someone stops by, it doesn't really matter what they talk about. If he likes the person, he just enjoys the time spent.
For Aspies, it's the content that counts. For me, content is nearly everything. Even if I'm just joking around with people at work, I'm paying attention to my words so that I get the right meaning across.
To me, there is nothing casual about conversation. I'm just not a casual person. I tend to be pretty serious a lot of the time. I think that's why I developed a good sense of humor--I had to have something to relieve the stress.