Four times in Five years is NOT normal. Not by any standards, and certainly not at your age. There is definitely something else going on.
I have a girlfriend who has not had sex with her husband now for over 6 years. They have a child who is 13. She has decided that she will stay with him until their child has graduated from college, and then she is leaving him. This is a deal breaker. She really likes her husband, in fact they are "best friends", but if you are young enough, it should be a normal part of union. She considers him a roommate now, and has had one affair, and keeps an open eye for other affairs.
Is that where you want to be? If you believe sex is a worthwhile part of marriage, then you need to nip this in the bud.
Have you been to a marriage counselor? Has he been to a doctor? Is he depressed? Does he sleep a lot, or come down with a long list of ailments?
My friend is sure that her husband suffers from depression, but he is unwilling to seek any help for this. She is a doctor, and is able to give him viagra, but he is unwilling to take this.
She is in really good shape. He is rather reserved, and she is wanton. She thinks that she is too aggressive for him. Also, he has a great distaste for bodily fluids.
We both have discussed whether he is gay, which unfortunately, is a big possibility.
Gosh, I don't mean to be negative, but there is something going on here. No one should have a sexless marriage until they are seniors, and then, people do still have sex. It has ZERO to do with morality. Having sex 4 times in 5 years does not make you immoral. Actually, it is a form of abuse. Yes, I really mean that. Morality is simply a term used by the repressed religious types and puritans. Sex is a normal human biological fact. It is only humans that put a label on it, and it is American's who take to a new level of puritan ethic. Please do not even go there.
But then is the second paragraph you talk about getting a lover, which is a huge switch from the prior paragraph. If you want to stay with your husband, and he is absolutely against having sex, and would not be opposed to you getting a lover, then why not? But there are pitfalls that are hard to ignore. For one thing, no matter the circumstance, it is icky. Women have a hard time separating their love from lovers. It is part of our instincts. Men can do it very well, women cannot. It will make things very complicated.
You really need to get this out in the open. You didn't say what he said when you "talked about it"? You say there are no secrets, but there has to be. If you have been open with him and say that you want more sex, and he has not supplied that for you, then there is something being held back.
You really haven't given this forum enough information to make a good value judgment, but hopefully we have given you some food for thought. I'm so sorry you have this challenge ahead of you, it is a very hard one. Eventually you will have to make a decision about what you want in a marriage. You are still very young.
Feel free to elaborate, and we will be there for you. But in the end, a forum is not going to completely help you, it sounds like both of you need to get to a therapist a.s.a.p.
Good luck.
Molly