Is retaliating to threats cled as self defense?

No. that is not sufficient to be considered as self defense. He has to actually attack you, or make attempts to attack to. If he tried hitting you with something then yes you could say that. Right now you could only say that he's harrasing you.