Cheese causes me problems too. 1. because the casein protein causes super insulin spikes (sometimes more than carbs do). 2. because it aggravates candida.
Calcium is also poorly absorbed from dairy sources in my research.
I just take a calcium citrate supplement with as few fillers as I could find.