You came to the truth in your post without stating it...
the more complexed (various forms of Sugar naturally associated with minerals and other co-factors) the Sugar is, the more "beneficial" it can be for us.
In today's world, people consume WAY too much Sugar to begin with... simple, refined sugars for the most part... bad.
Even if the sugars consumed by the average person were adequately complexed with minerals and co-factors, the average consumption rate per person is still WAY too high.