I think you're worrying too much; if you really want to co-author the novel with him, then do so. However, there is some validity in your thoughts, as it's likely to be difficult to establish a "loyal" fan base if you didn't write your debut alone
My take on it is you should write the novel yourself - it's not that hard, and stylistic differences in a work are painful - but ask for input/ideas from him. It's definitely a safer bet career-wise. If you're just doing it for the art though, you should just do whatever the hell you feel like doing and not worry about anything.
The page won't allow me to post links since I don't have enough posts. "Doing it for the art" is a reference. Hit up tvtropes dot org and search for "doing it for the art" if you care.