What might be happening is that the olive oil does actually "draw out" unhealthy sebum from your pores. More likely, however, is what you suggested--that olive oil is both good and bad. It has been used for thousands of years all around the world to beautify skin. It definitely has some beneficial chemicals (lauric acid, for example). As long as you remove it before the yeast can begin to feed on it, you should be okay. It's fantastic that this method is working for you!
I haven't started using your method yet, because right now I'm doing the following: I do not wash my face at all, with anything. I use "Clean & Clear Absorbing Sheets" to get rid of excess sebum, and I'm getting serious about getting healthy. I really think my gut is in bad shape, and that is the cause of my SD. Consider this--I have SD not only on my face and scalp, but also chest, back, ears, and LEGS. I have to believe this is not just a skin issue. Treating the skin (as you do) might be helpful, perhaps by removing yeast and helping the skin to "breathe," but I think the root cause of SD, at least in my case, is that there is something wrong with my stomach/intestines. I had a ton of ice cream 2 days ago and yesterday all my SD got much worse. My body's pH might be off, so I'm gonna start eating a teaspoon of baking soda each day, but more importantly I'm going to start eating a healthy diet, staying warm, and getting enough sleep. I think SD is an indication that my body (immune system) is not healthy.
I'll report back to let you know this method is doing for me.