If this was a continuous thing and I was being stalked, it would be different, but it's like wanting to kill somebody because they wink at you. If you don't want them to, tell them to stop, and if they don't, then you can call it sexual harassment. Harassment implies it's pressuring and continuous.
I'm not saying it's particularly right. A gentleman wouldn't be that forceful, and as a male myself, I know I wouldn't do anything of that nature to anybody, and I truly do feel sorry for whatever it is that happened in your situation. You and no other woman can automatically tell whether a comment is sexual or not, though. That's complete rubbish. If anything, it's a matter of assumption. That's still not your fault, though. It's society's fault that everything is sexualised.
Again, I feel that any man (or woman) who does that should keep it to themselves, but all I'm saying is that people need to lighten up about it unless it's legitimate harassment. The people in this thread have been legitimately harassed. I was not directing my comment at anybody in particular.