You seem like its possible that you might have Stockholm Syndrome, I recommend studying it. It might be that your inability to cope with the horror you endured as a child, the sick, twisted, corrupted situation that you had no support for and couldn't get away from, that really children can't handle, set your mind into a defense state called "stockholm syndrome" where you have unusually high feelings of sympathy and pity for your abusers that mask your true feelings. Its normal, I've seen it a lot, but please understand this might be effecting you, and inhibiting your recovery.
Its sort of a form of 'denial' defense, a sort of dissociation. It gives you a means of coping when you are overwhelmed and dont have what you need. sexua| abuse, especially to a child, is an Extremely disturbing experience, incredibly disruptive to the human organism. And if its from a parent, that can add a special additional complication and challenge.
Also, I don't think this man is going to disappear nicely. He will probably always be a problem unless forceably dealt with. Its obvious his conscience is completely dysfunctional if he has one at all.