If 4 doctors have said no I think you should just accept it. Be thankful for little things like having your physical health, I mean if public transport is a problem, there are more commuter friendly cities.
My worry is more the harm you could do to another. I myself was hit, as a pedestrian, by a driver who shouldn't have been driving. I was in a coma for 5 days. I was 15. My Dx was traumatic brain injury, traumatic brain stem injury, traumatic head injury and basilar skull fracture. I couldn't walk, talk, read or write. I went from the high school honor roll to special education. It took years to recover and I was never fully myself again.
Not driving is not the end of the world. I lived in downtown SF for a year and never needed a car and thought it was great. If there's a chance you could harm someone else, just don't do it.
I remember years ago going to a wedding, I had 2 glasses of wine, maybe 3. When I drove home I was a little buzzed and it was a thunderstorm. I thought, I could kill someone, so I pulled off and parked at a 7-11, got a coffee and just sat in my car for a couple hours before continuing home. I don't know what else to say.