The reason you may have trouble with milk is because it's not raw.
Anyway, thanks for your story, it will be interesting looking at this diet more. I'm only going to Aajonus Vonderplanitz at this time.
I believe that the paleo diet might not be as beneficial as it could be unless you first fix the digestion, either with a fruitarian type diet and/or with water-only fasting, both of which Aajonus did.
PS. I had Candida for 10 years and I recently overcame it, with a fruitarian diet btw.