I'm the same. I love my family dearly, hug them, touch them. Sexual touching I do not mind at all. It's friends, people, strangers and beauty therapists I don't like hugging, touching, sqyeezing, massaging me. Let's put it this way. For me a gift voucher for a massage is a definite no-no. It is almost painful. I end up with goosebumps, and shiver.
I don't think it is to do with upbringing. My mother was/is very loving to me and people. When I was growing up, there was a lot of kissing in the family. Traditionally, always kiss on the cheek when meet and goodbyes, on the daily basis. My mother's reasoning was, anything can happen, we may never see each other again.
So really don't know where it comes from. While being young, I would wince and bear it, but as I get older, sometimes I find myself even stepping back, just a bit. I don't want to do this, I feel ashamed. I mean, here zI am, could be having a great conversation with a person, but no touch. Strange! I can honestly say, there is nothing in past that I could put a blame on for this weird reaction.