You can eat anytime, even though this isn't the optimal way to go. The whole idea is not to swallow that bacteria that built up the night before. We can't prevent swallowing that bad stuff of course, but try to mitigate the process by swishing before you eat or drink anything.
I've swished in the morning and before dinner many times if I feel the need. The reason I rarely swish after eating is that it does seem to affect gallbladder production, and has been known to cause a tummy ache. But really, this is not a hard and fast rule by any means.
When you think about it, swishing after eating might not be such a bad idea as you want to clean your teeth effectively. Perhaps it might be a good idea that instead of an after dinner brandy, you take a shot glass of oil instead. And after you can throw your shot glass into the fire with David Niven style flair :-)