I'm thinking that you really are not "soul mates", whatever that phrase means.
It sounds more like a prisoner dreaming of escape from a cage! That sort of activity leads to a lot of mind tricks that cloud our thinking. Rationally, you have little to regret except the feeling of having missed out on the great thing that you had going with #1. One mind trick that you are playing is assuming that he would have been the ideal mate, nice family, great sex, live happily ever after, etc.
He has had 2 wives and managed to mess up with both of them, what makes you think that he/you wouldn't have suffered the same fate together?
I think that it is normal and adult for us to have regrets about missed opportunities. We are, after all, supposed to look out for our own best interests most of the time. When we make a choice and it turns out to be a bad one [or even a not optimum one], we inevitably spend some time thinking about that turning point and what it would have been like if we had taken the other road.
In our world of "no-fault" divorces, a lot of people try to jump off of the train and take the other route because it seems better than the one they are on. Some do it numerous times! I don't know of anyone who would counsel you to follow their lead. One of my colleagues [now on #4] has this advice for anyone who will listen: "Keep the first one. Make it work. Never give up." Good advice!
I love his wife[#1]like a sister and never have figured out why he divorced her for some hotter panties. Didn't make sense then, doesn't now! Finally, he agrees! Too late!
One other observation: I'm 72 and my wife of 50+ years can still make me roll over and beg for more. I used to set the pace, then age and Arthritis slowed me down a bit. She still turns me on like a light when she wants to. Have you ever thought of that?