If this younger man loved you and saw you as part of his future he would also try to accept your child and have a relationship with him. IMHO it is not fair to your son to carry on this relationship, it is also not fair to yourself because you will feel constantly torn as to whom you will show your affections and spend your time with.