I think after a lot of research that what is bad for us is not the milk but the pasteurization/homoginization processes. You can find out about it if you look at the raw milk sites. Nina Planck does an excellent job of describing it. And there is a ton of stuff on mercola.com.
Actually a lot of people like to use the argument about "we are the only species that drinks the milk of another species..." -- well, we are not. Orphaned animals that are adopted by "mothers" of other species will nurse them. Puppies used to nurse wet nurses to keep their milk production going in between human babies! Raw goat or cow's milk is a great source of nutrients, high-quality fats and other goodness. But you might want to re-think dairy that has been processed. I started just buying organic for my babies, but now I have found a raw-milk supplier. Actually raw goat milk is like a health food in and of itself!