I think he didn't want to get caught up in another scandal or more gossip (even if there is no truth to it). People were already starting to talk about the two of them having an affair or the trouble he would get into with Kristen. His career is on a high now - he can't mess it up. I also think they look odd together - he is too old for her. She looks much younger than Blake Lively who he looked ok with.