What is wrong with me?

I understand but I still feel uncomfortable about it. It isn't fair because I shouldn't just say you can't come to see your friend. It's just the history and my insecurities I guess.