Sounds like it has a lot to do with give and take.
If your not willing to "take it", something else is going to have to give, that would be him.
One suggestion I can offer is decide what you want and don't settle for less.
When you do that something somewhere is going to give, because you will be exerting influence toward that end.
I've seen many relationships flow on the basis of - who is the most determined toward what they want.
Right now it sounds like that would be him, but that can change just as sure as the earth revolves around the sun.