Don't know if this will help...yet, but I look at the 'screen name' you have chosen, 'love my babies'.
That says worlds about your mental health! Good for you!
Also, you and your husband are working well together. Good for you!
Somewhere are people who experienced happy lives. They have patterned themselves after their role models, and they are safe because they automatically take normal precautions. They don't have to 'think' about it.
1... You have discussed this with your husband. You don't 'have to' tell anyone else. Adults don't have to give 'reasons' for anything, just be mannerly. (Your memories don't show on your face.)
2... Your memories are lodged only in your tissues. They are just echoes. And, they can be replaced. Isn't that neat?!
(Brain fibers get little bumps which travel up the fibers the more they are used; and travel DOWN the fibers the less they are used. The bumps are created by thoughts. And we have millions of them.)
So, inwardly, think "No", or "Cancel", or "Applesauce", whenever you need to. And add some thoughts you really like. As surely as God made little green apples, you will improve in the direction you like better!
(This may sound silly, in your head, the first few times you try it. Good. Then you can laugh, out loud. The physical mechanics of laughing, heal, even when there is nothing funny going on. Try it.)
Outwardly, find people who are happy and doing things they love, and stick to them like glue! Take your wonderful family with you. Everyone will benefit.
Trust yourself. You've got everything you need, within you, to build the happiest of lives...of marriages, and of children.