Honestly, I think it might be a bit rude, depending on the situation. If you're out somewhere in public, and someone comes up to ask a question, it's probably not rude to leave one earbud in, and take one out, but you don't have to fold them up and put them in your pocket probably. BUT, if you're in someone's home or office, restaurant, or certainly in class or at work, I think earbuds belong in the pocket.
The issue isn't so much, whether or not you can hear and communicate with the other person while still having an earbud or listening to something; it's how it feels to the other person, who may not know whether or not they have your attention.
Charles