I agree with you in that dairy is not suitable for people despite what is believed by the general population. After what I've read up on dairy I've learnt to stay away from it and have actually stopped drinking milk for much longer than I have started the candida diet, although cheese as well as protein shakes was big on my favourite foods list which may have contributed to my acne.
For the longest time I've wanted to eat fruit but I'm constantly afraid that the Sugar in fruits may feed the candida, even if it may not be a lot, I'd rather rid of the candida overgrowth faster.
As for the olive oil, I'm actually using extra virgin olive oil if it makes a difference in how natural it is. If its still not, would substituting extra virgin olive oil for coconut oil be more acceptable?