I've had these issues. I will be in a conversation, and listening to the person, and it'll be fine, but then all of a sudden what the person says will make no sense, and when I repeat it back the other person is like, WTF? I've had this with my dad as well, and he scolds me for it, saying if I had been paying attention to the context--and I had been paying attention to the conversation!--then I wouldn't have come back with such a wild interpretation.
I've noticed I also have delayed processing at times, where it's almost like the phonemes the other person says get stored in a buffer in short-term memory for processing after a few seconds; I will then repeat to myself exactly what was said and process the semantic meaning immediately from that.
Both issues transfer to Japanese, my second language I began learning as an adult. At first I blamed it on its being a second language, but I've noticed that I have both issues, with roughly the same patterns and outcomes (e.g. there are times when listening to someone speak Japanese, that, again, the phonemes seem to get put in a buffer and then when I repeat it to myself, I understand its semantic meaning immediately). I will probably never overcome these issues in either language, because I've had this issue in English, my native language, as an adult.