Fear of driving is understandable. I try to avoid highways because of the speed, timing of exits, inexplicably uncooperative drivers and just how tiring it is to manage all that. I used to get lost so many times and have something that may have looked like a meltdown in earlier years...
Driving around town is better and more familiar territory helps. The worry still exists, but at a much lower level. It's important to start small, build from there and take note of how much you can handle at a time.