I can't say that numbers are easier for me, but I recognize what you are saying about not being able to hold letters (or numbers) in working memory.
If people tell me a phone number, I invariably have to ask them to repeat it slowly. I can usually remember the first three digits but completely lose everything as the numbers pile up. The same goes for spelling things out. I'm like a kid. If someone wants to keep me from knowing what they are talking about, all they have to do is spell it out rather than saying the words.
_________________
Never let the weeds get higher than the garden,
Always keep a sapphire in your mind.
(Tom Waits "Get Behind the Mule")