I had a really hard time with that, myself.
If they say something that seems out of character, that they normally don't say, it may be a joke.
If they tell you to do something unusual, that doesn't make sense to you, it may be a joke.
For example, if somebody is sharing treats, and some falls on the floor, and they tell you they are all out and you should get yours off the floor, that's unusual. And it's a joke at your expense.
If someone asks you out on a date and tells you to meet them somewhere weird, it's probably a joke.
If they say the LOVE something you have heard them say they hate, it's a joke.
Use logic and analyze. A remark that doesn't fit is probably a joke. Ask yourself if it makes sense.