Oh? Oops, you were joking

It's hard enough for me to understand when someone is being sarcastic, rhetorical or making a joke when I'm looking right at them. I find it impossible for me to understand the fore mentioned when I'm talking to someone on the phone.