I understand ... I just don't feel too comfortable when someone is around me all the time during shopping, even if it's helpful and especially if it's not.
if you have your own style ... if you're know what you want, what you need, what looks good on you (and even more important - what doesn't) and if you're well informed - you just don't need an assistant.
however, if you think you need it then it has to be someone that you feel comfortable with so, yes, I think it is a time to break up with your SA ... just try not to hurt his feelings (or reputation, if possible).