But the thing is. I don't know of a single person that wants to go back...do you? So, someone is forcing this to happen against the people's will. And others may not be reading the fruit correctly and are being sucked into following it as though God was leading them to do so.
Edit: (Of course I'm saying this only if that was the true scenario)