Is there anything wrong with my self esteem?

It sounds like you have enough self esteem to protect yourself and that is a good thing. If you had no self esteem, you would throw yourself at guys like a sleaze. You have the sense to be aware of feelings and rudeness, another good quality not everyone has. Maybe you could practice diffusing your defensive responses, by saying something like, "I could meet you for a cup of coffee..." and playing it like its not a big date thing and think of it as an opportunity to make a friend. And that puts the ball in your court...a place and method of going out that you determine. And if then if you like them as a friend, and they ask you again, maybe you will be less likely to dismiss it.