In-Call Voice Problem

When someone calls me (or I call someone), I wouldn't be able to hear them unless I'm in a quiet room, like a closet. The person says he can hear me clearly, but I can barely hear the person. What's the issue and how can I fix this?